{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tutorial 4: Interpretation Tutorial (requires GPU)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../../imgs/lightautoml_logo_color.png\" alt=\"LightAutoML logo\" style=\"width:100%;\"/>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Official LightAutoML github repository is [here](https://github.com/sb-ai-lab/LightAutoML)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some of HTML static content is not loading, to solve this problem you can use [nbviewer](https://nbviewer.jupyter.org). Link on tutorial on nbviewer [here](https://nbviewer.org/github/sb-ai-lab/LightAutoML/blob/master/examples/tutorials/Tutorial_4_NLP_Interpretation.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last years deep neural networks / gradient boosting / ensembles of models allow to improve the soulution quality of many application task in field of natural language processing (NLP). The indicators of this improvement describe the partial behavior of the model and can hide errors, for example, errors in the construction of the model, errors in data collection. All this can be critical in tasks related to the processing of medical, forensic, banking data.\n",
    "In this tutorial we will check the NLP interpretation module of automl."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download library and make some imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !pip install lightautoml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import shutil\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from sklearn.metrics import roc_auc_score, mean_squared_error\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "from lightautoml.automl.presets.text_presets import TabularNLPAutoML\n",
    "from lightautoml.tasks import Task\n",
    "\n",
    "from lightautoml.addons.interpretation import LimeTextExplainer, L2XTextExplainer\n",
    "\n",
    "import transformers\n",
    "transformers.logging.set_verbosity(50)\n",
    "\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this tutorial we will use train dataset (train.csv) from [Jigsaw-Toxic-Comment-Classification-Challage](https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/overview). The dataset contains  textual comments and 6 attributes of this text (toxic, serve_toxic, obscene, treat, insult, identity_hate). For now, we will use only <i>toxic</i> attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>comment_text</th>\n",
       "      <th>toxic</th>\n",
       "      <th>severe_toxic</th>\n",
       "      <th>obscene</th>\n",
       "      <th>threat</th>\n",
       "      <th>insult</th>\n",
       "      <th>identity_hate</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0000997932d777bf</td>\n",
       "      <td>Explanation\\nWhy the edits made under my usern...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>000103f0d9cfb60f</td>\n",
       "      <td>D'aww! He matches this background colour I'm s...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>000113f07ec002fd</td>\n",
       "      <td>Hey man, I'm really not trying to edit war. It...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0001b41b1c6bb37e</td>\n",
       "      <td>\"\\nMore\\nI can't make any real suggestions on ...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0001d958c54c6e35</td>\n",
       "      <td>You, sir, are my hero. Any chance you remember...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>159566</th>\n",
       "      <td>ffe987279560d7ff</td>\n",
       "      <td>\":::::And for the second time of asking, when ...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>159567</th>\n",
       "      <td>ffea4adeee384e90</td>\n",
       "      <td>You should be ashamed of yourself \\n\\nThat is ...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>159568</th>\n",
       "      <td>ffee36eab5c267c9</td>\n",
       "      <td>Spitzer \\n\\nUmm, theres no actual article for ...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>159569</th>\n",
       "      <td>fff125370e4aaaf3</td>\n",
       "      <td>And it looks like it was actually you who put ...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>159570</th>\n",
       "      <td>fff46fc426af1f9a</td>\n",
       "      <td>\"\\nAnd ... I really don't think you understand...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>159571 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                      id                                       comment_text  \\\n",
       "0       0000997932d777bf  Explanation\\nWhy the edits made under my usern...   \n",
       "1       000103f0d9cfb60f  D'aww! He matches this background colour I'm s...   \n",
       "2       000113f07ec002fd  Hey man, I'm really not trying to edit war. It...   \n",
       "3       0001b41b1c6bb37e  \"\\nMore\\nI can't make any real suggestions on ...   \n",
       "4       0001d958c54c6e35  You, sir, are my hero. Any chance you remember...   \n",
       "...                  ...                                                ...   \n",
       "159566  ffe987279560d7ff  \":::::And for the second time of asking, when ...   \n",
       "159567  ffea4adeee384e90  You should be ashamed of yourself \\n\\nThat is ...   \n",
       "159568  ffee36eab5c267c9  Spitzer \\n\\nUmm, theres no actual article for ...   \n",
       "159569  fff125370e4aaaf3  And it looks like it was actually you who put ...   \n",
       "159570  fff46fc426af1f9a  \"\\nAnd ... I really don't think you understand...   \n",
       "\n",
       "        toxic  severe_toxic  obscene  threat  insult  identity_hate  \n",
       "0           0             0        0       0       0              0  \n",
       "1           0             0        0       0       0              0  \n",
       "2           0             0        0       0       0              0  \n",
       "3           0             0        0       0       0              0  \n",
       "4           0             0        0       0       0              0  \n",
       "...       ...           ...      ...     ...     ...            ...  \n",
       "159566      0             0        0       0       0              0  \n",
       "159567      0             0        0       0       0              0  \n",
       "159568      0             0        0       0       0              0  \n",
       "159569      0             0        0       0       0              0  \n",
       "159570      0             0        0       0       0              0  \n",
       "\n",
       "[159571 rows x 8 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# train.csv file from\n",
    "# https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/overview\n",
    "\n",
    "data = pd.read_csv('train.csv')\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Usage of AutoML "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use standard ``lightautoml.automl.presets.text_presets.TabularNLPAutoML`` preset with finetuned TinyBERT from [Hugging Face](https://huggingface.co)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "\n",
    "train, test = train_test_split(data, test_size=0.2, random_state=42)\n",
    "\n",
    "roles = {\n",
    "    'text': ['comment_text'], \n",
    "    'drop': ['id', 'severe_toxic', 'obscene', 'threat', 'insult', 'identity_hate'],\n",
    "    'target': 'toxic'\n",
    "}\n",
    "\n",
    "task = Task('binary')\n",
    "\n",
    "automl = TabularNLPAutoML(\n",
    "    task=task,\n",
    "    timeout=3600,\n",
    "    cpu_limit=1,\n",
    "    gpu_ids='0',\n",
    "    general_params={\n",
    "        'nested_cv': False,\n",
    "        'use_algos': [['nn']]\n",
    "    },\n",
    "    autonlp_params={\n",
    "        'sent_scaler': 'l2'\n",
    "    },\n",
    "    text_params={\n",
    "        'lang': 'en',\n",
    "        'bert_model': 'prajjwal1/bert-tiny'\n",
    "    },\n",
    "    nn_params={\n",
    "        'opt_params': {'lr': 1e-5},\n",
    "        'max_length': 128,\n",
    "        'bs': 32,\n",
    "        'n_epochs': 7,\n",
    "    }\n",
    "\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11:22:30] Stdout logging level is DEBUG.\n",
      "[11:22:30] Model language mode: en\n",
      "[11:22:30] Task: binary\n",
      "\n",
      "[11:22:30] Start automl preset with listed constraints:\n",
      "[11:22:30] - time: 3600.00 seconds\n",
      "[11:22:30] - CPU: 1 cores\n",
      "[11:22:30] - memory: 16 GB\n",
      "\n",
      "[11:22:30] \u001b[1mTrain data shape: (127656, 8)\u001b[0m\n",
      "\n",
      "[11:22:30] Layer \u001b[1m1\u001b[0m train process start. Time left 3599.85 secs\n",
      "[11:22:31] Start fitting \u001b[1mLvl_0_Pipe_0_Mod_0_TorchNN\u001b[0m ...\n",
      "[11:22:31] Training params: {'bs': 32, 'num_workers': 1, 'max_length': 128, 'opt_params': {'lr': 1e-05}, 'scheduler_params': {'patience': 5, 'factor': 0.5, 'verbose': True}, 'is_snap': False, 'snap_params': {'k': 1, 'early_stopping': True, 'patience': 1, 'swa': False}, 'init_bias': True, 'n_epochs': 7, 'input_bn': False, 'emb_dropout': 0.1, 'emb_ratio': 3, 'max_emb_size': 50, 'bert_name': 'prajjwal1/bert-tiny', 'pooling': 'cls', 'device': device(type='cuda', index=0), 'use_cont': True, 'use_cat': True, 'use_text': True, 'lang': 'en', 'deterministic': False, 'multigpu': False, 'random_state': 42, 'path_to_save': None, 'verbose_inside': None, 'verbose': 1, 'device_ids': None, 'n_out': 1, 'cat_features': [], 'cat_dims': [], 'cont_features': [], 'cont_dim': 0, 'text_features': ['concated__comment_text'], 'bias': array([[-2.24401446]])}\n",
      "[11:22:31] ===== Start working with \u001b[1mfold 0\u001b[0m for \u001b[1mLvl_0_Pipe_0_Mod_0_TorchNN\u001b[0m =====\n",
      "[11:22:36] number of text features: 1 \n",
      "[11:22:36] number of categorical features: 0 \n",
      "[11:22:36] number of continuous features: 0 \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train (loss=0.257356): 100%|██████████| 2660/2660 [02:12<00:00, 20.13it/s]\n",
      "val: 100%|██████████| 1330/1330 [01:07<00:00, 19.83it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11:25:59] Epoch: 0, train loss: 0.25735557079315186, val loss: 0.19599375128746033, val metric: 0.9640350800072578\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "train (loss=0.168968): 100%|██████████| 2660/2660 [02:09<00:00, 20.61it/s]\n",
      "val: 100%|██████████| 1330/1330 [01:04<00:00, 20.58it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11:29:13] Epoch: 1, train loss: 0.16896754503250122, val loss: 0.14401142299175262, val metric: 0.9713461808486132\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "train (loss=0.131891): 100%|██████████| 2660/2660 [02:09<00:00, 20.49it/s]\n",
      "val: 100%|██████████| 1330/1330 [01:03<00:00, 20.87it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11:32:26] Epoch: 2, train loss: 0.1318911910057068, val loss: 0.12361849099397659, val metric: 0.9742718921629787\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "train (loss=0.114705): 100%|██████████| 2660/2660 [02:07<00:00, 20.90it/s]\n",
      "val: 100%|██████████| 1330/1330 [01:04<00:00, 20.76it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11:35:38] Epoch: 3, train loss: 0.11470535397529602, val loss: 0.11394938081502914, val metric: 0.9763582643756192\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "train (loss=0.103179): 100%|██████████| 2660/2660 [02:09<00:00, 20.54it/s]\n",
      "val: 100%|██████████| 1330/1330 [01:05<00:00, 20.36it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11:38:53] Epoch: 4, train loss: 0.10317856818437576, val loss: 0.10656153410673141, val metric: 0.9775081138714583\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "train (loss=0.0965996): 100%|██████████| 2660/2660 [02:09<00:00, 20.49it/s]\n",
      "val: 100%|██████████| 1330/1330 [01:05<00:00, 20.24it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11:42:08] Epoch: 5, train loss: 0.09659960865974426, val loss: 0.10427780449390411, val metric: 0.9783243683208365\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "train (loss=0.090561): 100%|██████████| 2660/2660 [02:11<00:00, 20.24it/s] \n",
      "val: 100%|██████████| 1330/1330 [01:02<00:00, 21.23it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11:45:22] Epoch: 6, train loss: 0.09056100249290466, val loss: 0.10337436944246292, val metric: 0.9788043902058639\n",
      "[11:45:23] ===== Start working with \u001b[1mfold 1\u001b[0m for \u001b[1mLvl_0_Pipe_0_Mod_0_TorchNN\u001b[0m =====\n",
      "[11:45:28] number of text features: 1 \n",
      "[11:45:28] number of categorical features: 0 \n",
      "[11:45:28] number of continuous features: 0 \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train (loss=0.257485): 100%|██████████| 2660/2660 [02:04<00:00, 21.30it/s]\n",
      "val: 100%|██████████| 1330/1330 [01:04<00:00, 20.67it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11:48:38] Epoch: 0, train loss: 0.2574850618839264, val loss: 0.19478833675384521, val metric: 0.961936968917119\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "train (loss=0.170552): 100%|██████████| 2660/2660 [02:08<00:00, 20.72it/s]\n",
      "val: 100%|██████████| 1330/1330 [01:06<00:00, 19.87it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11:51:53] Epoch: 1, train loss: 0.1705523431301117, val loss: 0.1437842845916748, val metric: 0.970873732336761\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "train (loss=0.132485): 100%|██████████| 2660/2660 [02:05<00:00, 21.15it/s]\n",
      "val: 100%|██████████| 1330/1330 [01:03<00:00, 20.97it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11:55:03] Epoch: 2, train loss: 0.13248467445373535, val loss: 0.12127983570098877, val metric: 0.9751468710522353\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "train (loss=0.11448): 100%|██████████| 2660/2660 [02:03<00:00, 21.51it/s] \n",
      "val: 100%|██████████| 1330/1330 [01:02<00:00, 21.11it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[11:58:09] Epoch: 3, train loss: 0.11447965353727341, val loss: 0.11149459332227707, val metric: 0.9768346789459879\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "train (loss=0.103458): 100%|██████████| 2660/2660 [02:06<00:00, 20.99it/s]\n",
      "val: 100%|██████████| 1330/1330 [01:04<00:00, 20.65it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12:01:20] Epoch: 4, train loss: 0.10345754027366638, val loss: 0.10722416639328003, val metric: 0.9782435623593337\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "train (loss=0.0963441): 100%|██████████| 2660/2660 [02:05<00:00, 21.17it/s]\n",
      "val: 100%|██████████| 1330/1330 [01:03<00:00, 20.91it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12:04:30] Epoch: 5, train loss: 0.09634406119585037, val loss: 0.10441421717405319, val metric: 0.978748563376753\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "train (loss=0.0900231): 100%|██████████| 2660/2660 [02:05<00:00, 21.17it/s]\n",
      "val: 100%|██████████| 1330/1330 [01:03<00:00, 20.84it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12:07:39] Epoch: 6, train loss: 0.09002314507961273, val loss: 0.10312184691429138, val metric: 0.9791290354336872\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12:07:40] ===== Start working with \u001b[1mfold 2\u001b[0m for \u001b[1mLvl_0_Pipe_0_Mod_0_TorchNN\u001b[0m =====\n",
      "[12:07:44] number of text features: 1 \n",
      "[12:07:44] number of categorical features: 0 \n",
      "[12:07:44] number of continuous features: 0 \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train (loss=0.257448): 100%|██████████| 2660/2660 [02:04<00:00, 21.45it/s]\n",
      "val: 100%|██████████| 1330/1330 [01:00<00:00, 21.91it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12:10:50] Epoch: 0, train loss: 0.2574479281902313, val loss: 0.19449889659881592, val metric: 0.9648288318293945\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train (loss=0.169502): 100%|██████████| 2660/2660 [02:03<00:00, 21.52it/s]\n",
      "val: 100%|██████████| 1330/1330 [01:01<00:00, 21.79it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12:13:55] Epoch: 1, train loss: 0.1695016324520111, val loss: 0.14307956397533417, val metric: 0.9706200035841146\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train (loss=0.131626): 100%|██████████| 2660/2660 [02:03<00:00, 21.54it/s]\n",
      "val: 100%|██████████| 1330/1330 [01:00<00:00, 21.84it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12:16:59] Epoch: 2, train loss: 0.13162554800510406, val loss: 0.12111066281795502, val metric: 0.97454294780979\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train (loss=0.114015): 100%|██████████| 2660/2660 [02:03<00:00, 21.57it/s]\n",
      "val: 100%|██████████| 1330/1330 [01:00<00:00, 21.83it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12:20:04] Epoch: 3, train loss: 0.11401509493589401, val loss: 0.11131983995437622, val metric: 0.9763178957078734\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train (loss=0.104155): 100%|██████████| 2660/2660 [02:03<00:00, 21.56it/s]\n",
      "val: 100%|██████████| 1330/1330 [01:00<00:00, 21.87it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12:23:08] Epoch: 4, train loss: 0.10415521264076233, val loss: 0.10691472887992859, val metric: 0.9772204526836245\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train (loss=0.0953203): 100%|██████████| 2660/2660 [02:04<00:00, 21.41it/s]\n",
      "val: 100%|██████████| 1330/1330 [01:01<00:00, 21.67it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12:26:13] Epoch: 5, train loss: 0.09532025456428528, val loss: 0.10362745076417923, val metric: 0.9780747656394276\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train (loss=0.0899258): 100%|██████████| 2660/2660 [02:04<00:00, 21.34it/s]\n",
      "val: 100%|██████████| 1330/1330 [01:01<00:00, 21.68it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12:29:20] Epoch: 6, train loss: 0.08992581069469452, val loss: 0.10427321493625641, val metric: 0.9781931517871759\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "val: 100%|██████████| 1330/1330 [01:01<00:00, 21.53it/s]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12:30:21] Early stopping: val loss: 0.10362745076417923, val metric: 0.9780747656394276\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12:30:22] Fitting \u001b[1mLvl_0_Pipe_0_Mod_0_TorchNN\u001b[0m finished. score = \u001b[1m0.9782371823652668\u001b[0m\n",
      "[12:30:22] \u001b[1mLvl_0_Pipe_0_Mod_0_TorchNN\u001b[0m fitting and predicting completed\n",
      "[12:30:22] Time left -472.15 secs\n",
      "\n",
      "[12:30:22] Time limit exceeded. Last level models will be blended and unused pipelines will be pruned.\n",
      "\n",
      "[12:30:22] \u001b[1mLayer 1 training completed.\u001b[0m\n",
      "\n",
      "[12:30:22] \u001b[1mAutoml preset training completed in 4072.15 seconds\u001b[0m\n",
      "\n",
      "[12:30:22] Model description:\n",
      "Final prediction for new objects (level 0) = \n",
      "\t 1.00000 * (3 averaged models Lvl_0_Pipe_0_Mod_0_TorchNN) \n",
      "\n",
      "[12:30:22] number of text features: 1 \n",
      "[12:30:22] number of categorical features: 0 \n",
      "[12:30:22] number of continuous features: 0 \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "test: 100%|██████████| 998/998 [00:47<00:00, 21.08it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12:31:15] number of text features: 1 \n",
      "[12:31:15] number of categorical features: 0 \n",
      "[12:31:15] number of continuous features: 0 \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "test: 100%|██████████| 998/998 [00:46<00:00, 21.51it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12:32:08] number of text features: 1 \n",
      "[12:32:08] number of categorical features: 0 \n",
      "[12:32:08] number of continuous features: 0 \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "test: 100%|██████████| 998/998 [00:46<00:00, 21.47it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Check scores:\n",
      "OOF score: 0.9782371823652668\n",
      "TEST score: 0.9807740353486142\n",
      "CPU times: user 18min 47s, sys: 1min 15s, total: 20min 3s\n",
      "Wall time: 1h 10min 30s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "oof_pred = automl.fit_predict(train, roles=roles, verbose = 10)\n",
    "test_pred = automl.predict(test)\n",
    "not_nan = np.any(~np.isnan(oof_pred.data), axis=1)\n",
    "\n",
    "print('Check scores:')\n",
    "print('OOF score: {}'.format(roc_auc_score(train[roles['target']].values[not_nan], oof_pred.data[not_nan][:, 0])))\n",
    "print('TEST score: {}'.format(roc_auc_score(test[roles['target']].values, test_pred.data[:, 0])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "automl.set_verbosity_level(0) # refuse logging in automl"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## LIME"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Linear approximation of model nearby selected object. The weights of this linear model is feature attribution for automl's prediction for this object.\n",
    "\n",
    "Algorithm:\n",
    "\n",
    "0. Select object to interpret.\n",
    "\n",
    "1. Select the input text column, that will be explained (```perturb_column```). All other columns of object will be fixed.\n",
    "\n",
    "2. A dataset of size ```n_sample``` (by default ``5000``) is created by randomly deleting tokens (in groups). Dataset is binary (there is a token if one and no token if zero).\n",
    "\n",
    "3. Predict with AutoML module target values for created dataset.\n",
    "\n",
    "4. Optionally, the selection of features (important tokens) is performed using LASSO (```feature_selection='lasso'```, you can also ```'none'``` to not select and get them all). The number of features used after feature selection is ```n_feautres``` (```= 10``` by default).\n",
    "\n",
    "5. We train the explained model on this (a linear model with weights, the method of calculating weights is the cosine distance by default, you can also use your own function or the name of the distance from ```sklearn.metrics.pairwise_distances```).\n",
    "\n",
    "6. The weights of the linear model are the interpretation.\n",
    "\n",
    "P.S. Care about the sentence length. Detokenization works within $O(n^2)$, where $n$ -- sentence length. \n",
    "\n",
    "\n",
    "Scheme of work:\n",
    "\n",
    "\n",
    "<img src=\"../../imgs/lime.jpg\" alt=\"lime\" style=\"width:100%;\"/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# LimeTextExplainer for NLP preset\n",
    "lime = LimeTextExplainer(automl, feature_selection='lasso', force_order=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's try it on neutral text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "test: 100%|██████████| 157/157 [00:02<00:00, 77.34it/s]\n",
      "test: 100%|██████████| 157/157 [00:01<00:00, 79.52it/s]\n",
      "test: 100%|██████████| 157/157 [00:01<00:00, 78.82it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #FFE4E4;\">The</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFBABA;\">lyrics</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #E3E3FF;\">is</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFAAAA;\">found</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFCECE;\">in</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFE4E4;\">the</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #DCDCFF;\">German</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FF7E7E;\">version</span> <span style=\"background-color: #FFFEFE;\">, </span> <span style=\"background-color: #FFFEFE;\">so</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFEFE;\">I</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFA6A6;\">assume</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFF0F0;\">it</span> <span style=\"background-color: #FFFEFE;\">'</span> <span style=\"background-color: #FFFEFE;\">s</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFC6C6;\">usable</span> <span style=\"background-color: #FFFEFE;\"></span> <span style=\"background-color: #FFFEFE;\">. ~</span></p>\n",
       "        </div>\n",
       "        \n",
       "        <p style=\"text-align: center\">\n",
       "            Class mapping\n",
       "        </p>\n",
       "        <div style=\"margin-left: 10%; margin-right: 10%; \">\n",
       "            <div id=\"grad\" style=\"background: linear-gradient(90deg, #FF7E7E 0%, #FFFEFE 50%, #8080FF 100%); border: 1px solid black; margin-left: 5.0em; margin-right: 5.0em; float: center; \">\n",
       "                <p style=\"text-align:left; margin-left: 1%; margin-right: 1%; color: white;\">\n",
       "                    Class: 0\n",
       "                    <span style=\"float:right;\">\n",
       "                        Class: 1\n",
       "                    </span>\n",
       "                </p>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"margin-left: 5.0em; margin-right: 5.0em; \">\n",
       "                <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-6.1</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-4.9</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-3.6</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-2.4</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-1.2</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.0</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">1.2</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">2.4</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">3.6</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">4.9</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 0%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">6.1</span></div>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"float: right; right: 0.75em; top: -3em; position: relative; font-weight: bold;\">Scale</div>\n",
       "            <div style=\"float: right; right: -2em; top: -2.9em; position: relative; font-weight: bold;\">✕ 1e-05</div>\n",
       "\n",
       "            <div style=\"float: left; left: -5.5em; top: -4.42em; position: relative;  font-weight: bold;\">AutoML's prediction</div>\n",
       "            <div style=\"float: left; left: -8.22em; top: -2.9em; position: relative;  font-weight: bold;\">0.003</div>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "exp = lime.explain_instance(test.loc[34019], labels=(0, 1), perturb_column='comment_text')\n",
    "exp.visualize_in_notebook(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Toxic comments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "test: 100%|██████████| 157/157 [00:01<00:00, 93.42it/s]\n",
      "test: 100%|██████████| 157/157 [00:01<00:00, 92.47it/s]\n",
      "test: 100%|██████████| 157/157 [00:01<00:00, 93.75it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #DADAFF;\">A</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #9E9EFF;\">silly</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #8080FF;\">fat</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #A2A2FF;\">cow</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFCFC;\">who</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFE2E2;\">won</span> <span style=\"background-color: #FFFEFE;\">'</span> <span style=\"background-color: #FCFCFF;\">t</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FEFEFF;\">leave</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #F2F2FF;\">me</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FEFEFF;\">alone</span></p>\n",
       "        </div>\n",
       "        \n",
       "        <p style=\"text-align: center\">\n",
       "            Class mapping\n",
       "        </p>\n",
       "        <div style=\"margin-left: 10%; margin-right: 10%; \">\n",
       "            <div id=\"grad\" style=\"background: linear-gradient(90deg, #FF7E7E 0%, #FFFEFE 50%, #8080FF 100%); border: 1px solid black; margin-left: 5.0em; margin-right: 5.0em; float: center; \">\n",
       "                <p style=\"text-align:left; margin-left: 1%; margin-right: 1%; color: white;\">\n",
       "                    Class: 0\n",
       "                    <span style=\"float:right;\">\n",
       "                        Class: 1\n",
       "                    </span>\n",
       "                </p>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"margin-left: 5.0em; margin-right: 5.0em; \">\n",
       "                <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-2.3</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-1.8</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-1.4</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-0.9</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-0.5</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.0</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.5</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.9</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">1.4</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">1.8</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 0%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">2.3</span></div>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"float: right; right: 0.75em; top: -3em; position: relative; font-weight: bold;\">Scale</div>\n",
       "            <div style=\"float: right; right: -2em; top: -2.9em; position: relative; font-weight: bold;\">✕ 1e-01</div>\n",
       "\n",
       "            <div style=\"float: left; left: -5.5em; top: -4.42em; position: relative;  font-weight: bold;\">AutoML's prediction</div>\n",
       "            <div style=\"float: left; left: -8.22em; top: -2.9em; position: relative;  font-weight: bold;\">0.913</div>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "exp = lime.explain_instance(test.loc[78687], labels=(0, 1), perturb_column='comment_text')\n",
    "exp.visualize_in_notebook(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's see on uncertain expamples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "test: 100%|██████████| 157/157 [00:02<00:00, 71.46it/s]\n",
      "test: 100%|██████████| 157/157 [00:02<00:00, 71.48it/s]\n",
      "test: 100%|██████████| 157/157 [00:02<00:00, 71.15it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #FFEAEA;\">Why</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #ECECFF;\">are</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #AEAEFF;\">you</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFF2F2;\">still</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFF0F0;\">here</span> <span style=\"background-color: #FFFEFE;\">? </span> <span style=\"background-color: #FFE0E0;\">Can</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #AEAEFF;\">you</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFF6F6;\">not</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFE0E0;\">find</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #F6F6FF;\">anything</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFEEEE;\">more</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FF9E9E;\">important</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFF4F4;\">to</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFAFA;\">do</span> <span style=\"background-color: #FFFEFE;\">, </span> <span style=\"background-color: #E8E8FF;\">like</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #8080FF;\">killing</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #B8B8FF;\">yourself</span> <span style=\"background-color: #FCFCFF;\"></span> <span style=\"background-color: #FFFEFE;\">?</span></p>\n",
       "        </div>\n",
       "        \n",
       "        <p style=\"text-align: center\">\n",
       "            Class mapping\n",
       "        </p>\n",
       "        <div style=\"margin-left: 10%; margin-right: 10%; \">\n",
       "            <div id=\"grad\" style=\"background: linear-gradient(90deg, #FF7E7E 0%, #FFFEFE 50%, #8080FF 100%); border: 1px solid black; margin-left: 5.0em; margin-right: 5.0em; float: center; \">\n",
       "                <p style=\"text-align:left; margin-left: 1%; margin-right: 1%; color: white;\">\n",
       "                    Class: 0\n",
       "                    <span style=\"float:right;\">\n",
       "                        Class: 1\n",
       "                    </span>\n",
       "                </p>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"margin-left: 5.0em; margin-right: 5.0em; \">\n",
       "                <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-1.3</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-1.1</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-0.8</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-0.5</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-0.3</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.0</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.3</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.5</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.8</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">1.1</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 0%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">1.3</span></div>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"float: right; right: 0.75em; top: -3em; position: relative; font-weight: bold;\">Scale</div>\n",
       "            <div style=\"float: right; right: -2em; top: -2.9em; position: relative; font-weight: bold;\">✕ 1e-01</div>\n",
       "\n",
       "            <div style=\"float: left; left: -5.5em; top: -4.42em; position: relative;  font-weight: bold;\">AutoML's prediction</div>\n",
       "            <div style=\"float: left; left: -8.22em; top: -2.9em; position: relative;  font-weight: bold;\">0.107</div>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "exp = lime.explain_instance(test.loc[4733], labels=(0, 1), perturb_column='comment_text', n_features=20) \n",
    "exp.visualize_in_notebook(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's delete 'important' from this abstract. We can see that automl increase it's probability of toxicity of this abstract"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "test.loc[4733, 'comment_text'] = 'Why are you still here ? Can you not find anything more to do , like killing yourself ?'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "test: 100%|██████████| 157/157 [00:02<00:00, 73.97it/s]\n",
      "test: 100%|██████████| 157/157 [00:02<00:00, 73.07it/s]\n",
      "test: 100%|██████████| 157/157 [00:02<00:00, 73.22it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #FFECEC;\">Why</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #EEEEFF;\">are</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #B6B6FF;\">you</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFF0F0;\">still</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFF2F2;\">here</span> <span style=\"background-color: #FFFEFE;\"> ? </span> <span style=\"background-color: #FFE6E6;\">Can</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #B6B6FF;\">you</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFF4F4;\">not</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFDEDE;\">find</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #F8F8FF;\">anything</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFECEC;\">more</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFF2F2;\">to</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFAFA;\">do</span> <span style=\"background-color: #FFFEFE;\"> , </span> <span style=\"background-color: #ECECFF;\">like</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #8080FF;\">killing</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #BABAFF;\">yourself</span> <span style=\"background-color: #FFFEFE;\"></span> <span style=\"background-color: #FFFEFE;\"> ?</span></p>\n",
       "        </div>\n",
       "        \n",
       "        <p style=\"text-align: center\">\n",
       "            Class mapping\n",
       "        </p>\n",
       "        <div style=\"margin-left: 10%; margin-right: 10%; \">\n",
       "            <div id=\"grad\" style=\"background: linear-gradient(90deg, #FF7E7E 0%, #FFFEFE 50%, #8080FF 100%); border: 1px solid black; margin-left: 5.0em; margin-right: 5.0em; float: center; \">\n",
       "                <p style=\"text-align:left; margin-left: 1%; margin-right: 1%; color: white;\">\n",
       "                    Class: 0\n",
       "                    <span style=\"float:right;\">\n",
       "                        Class: 1\n",
       "                    </span>\n",
       "                </p>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"margin-left: 5.0em; margin-right: 5.0em; \">\n",
       "                <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-2.2</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-1.8</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-1.3</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-0.9</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-0.4</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.0</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.4</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.9</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">1.3</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">1.8</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 0%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">2.2</span></div>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"float: right; right: 0.75em; top: -3em; position: relative; font-weight: bold;\">Scale</div>\n",
       "            <div style=\"float: right; right: -2em; top: -2.9em; position: relative; font-weight: bold;\">✕ 1e-01</div>\n",
       "\n",
       "            <div style=\"float: left; left: -5.5em; top: -4.42em; position: relative;  font-weight: bold;\">AutoML's prediction</div>\n",
       "            <div style=\"float: left; left: -8.22em; top: -2.9em; position: relative;  font-weight: bold;\">0.270</div>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "exp = lime.explain_instance(test.loc[4733], labels=(0, 1), perturb_column='comment_text', n_features=20) \n",
    "exp.visualize_in_notebook(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we add the 'relability' the AutoML decrease the toxicity probability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "test.loc[4733, 'comment_text'] = 'Why are you still here ? Can you not find anything more to do , like killing yourself ? relability'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "test: 100%|██████████| 157/157 [00:02<00:00, 68.66it/s]\n",
      "test: 100%|██████████| 157/157 [00:02<00:00, 64.12it/s]\n",
      "test: 100%|██████████| 157/157 [00:02<00:00, 68.18it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #FFE8E8;\">Why</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #F0F0FF;\">are</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #ACACFF;\">you</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFEEEE;\">still</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFEEEE;\">here</span> <span style=\"background-color: #FFFEFE;\"> ? </span> <span style=\"background-color: #FFDCDC;\">Can</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #ACACFF;\">you</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFF4F4;\">not</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFDCDC;\">find</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FAFAFF;\">anything</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFE8E8;\">more</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFEEEE;\">to</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFF8F8;\">do</span> <span style=\"background-color: #FFFEFE;\"> , </span> <span style=\"background-color: #EEEEFF;\">like</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #8080FF;\">killing</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #BABAFF;\">yourself</span> <span style=\"background-color: #FFFEFE;\"> ? </span> <span style=\"background-color: #FCFCFF;\">relability</span></p>\n",
       "        </div>\n",
       "        \n",
       "        <p style=\"text-align: center\">\n",
       "            Class mapping\n",
       "        </p>\n",
       "        <div style=\"margin-left: 10%; margin-right: 10%; \">\n",
       "            <div id=\"grad\" style=\"background: linear-gradient(90deg, #FF7E7E 0%, #FFFEFE 50%, #8080FF 100%); border: 1px solid black; margin-left: 5.0em; margin-right: 5.0em; float: center; \">\n",
       "                <p style=\"text-align:left; margin-left: 1%; margin-right: 1%; color: white;\">\n",
       "                    Class: 0\n",
       "                    <span style=\"float:right;\">\n",
       "                        Class: 1\n",
       "                    </span>\n",
       "                </p>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"margin-left: 5.0em; margin-right: 5.0em; \">\n",
       "                <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-2.0</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-1.6</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-1.2</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-0.8</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-0.4</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.0</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.4</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.8</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">1.2</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">1.6</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 0%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">2.0</span></div>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"float: right; right: 0.75em; top: -3em; position: relative; font-weight: bold;\">Scale</div>\n",
       "            <div style=\"float: right; right: -2em; top: -2.9em; position: relative; font-weight: bold;\">✕ 1e-01</div>\n",
       "\n",
       "            <div style=\"float: left; left: -5.5em; top: -4.42em; position: relative;  font-weight: bold;\">AutoML's prediction</div>\n",
       "            <div style=\"float: left; left: -8.22em; top: -2.9em; position: relative;  font-weight: bold;\">0.231</div>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "exp = lime.explain_instance(test.loc[4733], labels=(0, 1), perturb_column='comment_text', n_features=20) \n",
    "exp.visualize_in_notebook(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "test: 100%|██████████| 157/157 [00:02<00:00, 57.57it/s]\n",
      "test: 100%|██████████| 157/157 [00:02<00:00, 56.72it/s]\n",
      "test: 100%|██████████| 157/157 [00:02<00:00, 56.36it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #F8F8FF;\">stop</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFF4F4;\">editing</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFEFE;\">this</span> <span style=\"background-color: #FFFEFE;\">, </span> <span style=\"background-color: #ECECFF;\">you</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #8888FF;\">dumbass</span> <span style=\"background-color: #FFFEFE;\">. </span> <span style=\"background-color: #FFF4F4;\">why</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFEFE;\">do</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #ECECFF;\">you</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFCFC;\">have</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFCFC;\">to</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FAFAFF;\">be</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFEFE;\">such</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFCFC;\">a</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #8080FF;\">bitch</span> <span style=\"background-color: #FFFEFE;\">? </span> <span style=\"background-color: #FFFCFC;\">the</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFAFA;\">ghosts</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFEFE;\">of</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FEFEFF;\">bill</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFEFE;\">maas</span> <span style=\"background-color: #FFFEFE;\">' </span> <span style=\"background-color: #FFFAFA;\">past</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFEFE;\">will</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFEFE;\">haunt</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #ECECFF;\">you</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFEFE;\">forever</span> <span style=\"background-color: #FFFEFE;\">!!! </span> <span style=\"background-color: #FFFCFC;\">MWAHAHHAHAA</span></p>\n",
       "        </div>\n",
       "        \n",
       "        <p style=\"text-align: center\">\n",
       "            Class mapping\n",
       "        </p>\n",
       "        <div style=\"margin-left: 10%; margin-right: 10%; \">\n",
       "            <div id=\"grad\" style=\"background: linear-gradient(90deg, #FF7E7E 0%, #FFFEFE 50%, #8080FF 100%); border: 1px solid black; margin-left: 5.0em; margin-right: 5.0em; float: center; \">\n",
       "                <p style=\"text-align:left; margin-left: 1%; margin-right: 1%; color: white;\">\n",
       "                    Class: 0\n",
       "                    <span style=\"float:right;\">\n",
       "                        Class: 1\n",
       "                    </span>\n",
       "                </p>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"margin-left: 5.0em; margin-right: 5.0em; \">\n",
       "                <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-2.8</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-2.3</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-1.7</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-1.1</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-0.6</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.0</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.6</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">1.1</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">1.7</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">2.3</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 0%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">2.8</span></div>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"float: right; right: 0.75em; top: -3em; position: relative; font-weight: bold;\">Scale</div>\n",
       "            <div style=\"float: right; right: -2em; top: -2.9em; position: relative; font-weight: bold;\">✕ 1e-01</div>\n",
       "\n",
       "            <div style=\"float: left; left: -5.5em; top: -4.42em; position: relative;  font-weight: bold;\">AutoML's prediction</div>\n",
       "            <div style=\"float: left; left: -8.22em; top: -2.9em; position: relative;  font-weight: bold;\">0.920</div>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "exp = lime.explain_instance(test.loc[40112], labels=(0, 1), perturb_column='comment_text', n_features=20) \n",
    "exp.visualize_in_notebook(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's delete the toxic words to 'good boy'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "test.loc[40112, 'comment_text'] = \"stop editing this, you good boy. why do you have to be such a good boy? the ghosts of bill maas' past will haunt you forever!!! MWAHAHHAHAA\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "test: 100%|██████████| 157/157 [00:02<00:00, 55.40it/s]\n",
      "test: 100%|██████████| 157/157 [00:02<00:00, 56.35it/s]\n",
      "test: 100%|██████████| 157/157 [00:02<00:00, 55.86it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #D8D8FF;\">stop</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFC3C3;\">editing</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFEFE;\">this</span> <span style=\"background-color: #FFFEFE;\">, </span> <span style=\"background-color: #8080FF;\">you</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FF8A8A;\">good</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #CACAFF;\">boy</span> <span style=\"background-color: #FFFEFE;\">. </span> <span style=\"background-color: #FFECEC;\">why</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFEFE;\">do</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #8080FF;\">you</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFF2F2;\">have</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFF2F2;\">to</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFEFE;\">be</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #F6F6FF;\">such</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #E6E6FF;\">a</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FF8A8A;\">good</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #CACAFF;\">boy</span> <span style=\"background-color: #FFFEFE;\">? </span> <span style=\"background-color: #FFFAFA;\">the</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #ECECFF;\">ghosts</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFF6F6;\">of</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #ECECFF;\">bill</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #F0F0FF;\">maas</span> <span style=\"background-color: #FFFEFE;\">' </span> <span style=\"background-color: #FFF8F8;\">past</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #F6F6FF;\">will</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FAFAFF;\">haunt</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #8080FF;\">you</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #ECECFF;\">forever</span> <span style=\"background-color: #FFFEFE;\">!!! </span> <span style=\"background-color: #FFF0F0;\">MWAHAHHAHAA</span></p>\n",
       "        </div>\n",
       "        \n",
       "        <p style=\"text-align: center\">\n",
       "            Class mapping\n",
       "        </p>\n",
       "        <div style=\"margin-left: 10%; margin-right: 10%; \">\n",
       "            <div id=\"grad\" style=\"background: linear-gradient(90deg, #FF7E7E 0%, #FFFEFE 50%, #8080FF 100%); border: 1px solid black; margin-left: 5.0em; margin-right: 5.0em; float: center; \">\n",
       "                <p style=\"text-align:left; margin-left: 1%; margin-right: 1%; color: white;\">\n",
       "                    Class: 0\n",
       "                    <span style=\"float:right;\">\n",
       "                        Class: 1\n",
       "                    </span>\n",
       "                </p>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"margin-left: 5.0em; margin-right: 5.0em; \">\n",
       "                <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-1.5</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-1.2</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-0.9</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-0.6</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-0.3</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.0</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.3</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.6</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.9</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">1.2</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 0%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">1.5</span></div>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"float: right; right: 0.75em; top: -3em; position: relative; font-weight: bold;\">Scale</div>\n",
       "            <div style=\"float: right; right: -2em; top: -2.9em; position: relative; font-weight: bold;\">✕ 1e-01</div>\n",
       "\n",
       "            <div style=\"float: left; left: -5.5em; top: -4.42em; position: relative;  font-weight: bold;\">AutoML's prediction</div>\n",
       "            <div style=\"float: left; left: -8.22em; top: -2.9em; position: relative;  font-weight: bold;\">0.157</div>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "exp = lime.explain_instance(test.loc[40112], labels=(0, 1), perturb_column='comment_text', n_features=20) \n",
    "exp.visualize_in_notebook(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try from neutral make toxic abstract."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "test: 100%|██████████| 157/157 [00:01<00:00, 101.90it/s]\n",
      "test: 100%|██████████| 157/157 [00:01<00:00, 100.88it/s]\n",
      "test: 100%|██████████| 157/157 [00:01<00:00, 99.18it/s] \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #D6D6FF;\">Okay</span> <span style=\"background-color: #FFFEFE;\">, </span> <span style=\"background-color: #FF7E7E;\">thanks</span> <span style=\"background-color: #FFFEFE;\">. </span> <span style=\"background-color: #FEFEFF;\">I</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFEFE;\">will</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FEFEFF;\">do</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FCFCFF;\">so</span> <span style=\"background-color: #FFFEFE;\"></span> <span style=\"background-color: #FFFEFE;\">.</span></p>\n",
       "        </div>\n",
       "        \n",
       "        <p style=\"text-align: center\">\n",
       "            Class mapping\n",
       "        </p>\n",
       "        <div style=\"margin-left: 10%; margin-right: 10%; \">\n",
       "            <div id=\"grad\" style=\"background: linear-gradient(90deg, #FF7E7E 0%, #FFFEFE 50%, #8080FF 100%); border: 1px solid black; margin-left: 5.0em; margin-right: 5.0em; float: center; \">\n",
       "                <p style=\"text-align:left; margin-left: 1%; margin-right: 1%; color: white;\">\n",
       "                    Class: 0\n",
       "                    <span style=\"float:right;\">\n",
       "                        Class: 1\n",
       "                    </span>\n",
       "                </p>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"margin-left: 5.0em; margin-right: 5.0em; \">\n",
       "                <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-1.2</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-1.0</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-0.7</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-0.5</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-0.2</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.0</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.2</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.5</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.7</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">1.0</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 0%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">1.2</span></div>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"float: right; right: 0.75em; top: -3em; position: relative; font-weight: bold;\">Scale</div>\n",
       "            <div style=\"float: right; right: -2em; top: -2.9em; position: relative; font-weight: bold;\">✕ 1e-03</div>\n",
       "\n",
       "            <div style=\"float: left; left: -5.5em; top: -4.42em; position: relative;  font-weight: bold;\">AutoML's prediction</div>\n",
       "            <div style=\"float: left; left: -8.22em; top: -2.9em; position: relative;  font-weight: bold;\">0.003</div>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "exp = lime.explain_instance(test.loc[18396], labels=(0, 1), perturb_column='comment_text', n_features=20) \n",
    "exp.visualize_in_notebook(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "test.loc[18396] = \"Okay , thanks . I will do so . dumbass please\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "test: 100%|██████████| 157/157 [00:01<00:00, 89.68it/s]\n",
      "test: 100%|██████████| 157/157 [00:01<00:00, 90.71it/s]\n",
      "test: 100%|██████████| 157/157 [00:01<00:00, 90.35it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #FFFAFA;\">Okay</span> <span style=\"background-color: #FFFEFE;\"> , </span> <span style=\"background-color: #FFF4F4;\">thanks</span> <span style=\"background-color: #FFFEFE;\"> . </span> <span style=\"background-color: #FEFEFF;\">I</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FCFCFF;\">will</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FEFEFF;\">do</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFEFE;\">so</span> <span style=\"background-color: #FFFEFE;\"> . </span> <span style=\"background-color: #8080FF;\">dumbass</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFAFA;\">please</span></p>\n",
       "        </div>\n",
       "        \n",
       "        <p style=\"text-align: center\">\n",
       "            Class mapping\n",
       "        </p>\n",
       "        <div style=\"margin-left: 10%; margin-right: 10%; \">\n",
       "            <div id=\"grad\" style=\"background: linear-gradient(90deg, #FF7E7E 0%, #FFFEFE 50%, #8080FF 100%); border: 1px solid black; margin-left: 5.0em; margin-right: 5.0em; float: center; \">\n",
       "                <p style=\"text-align:left; margin-left: 1%; margin-right: 1%; color: white;\">\n",
       "                    Class: 0\n",
       "                    <span style=\"float:right;\">\n",
       "                        Class: 1\n",
       "                    </span>\n",
       "                </p>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"margin-left: 5.0em; margin-right: 5.0em; \">\n",
       "                <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-8.0</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-6.4</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-4.8</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-3.2</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-1.6</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.0</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">1.6</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">3.2</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">4.8</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">6.4</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 0%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">8.0</span></div>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"float: right; right: 0.75em; top: -3em; position: relative; font-weight: bold;\">Scale</div>\n",
       "            <div style=\"float: right; right: -2em; top: -2.9em; position: relative; font-weight: bold;\">✕ 1e-01</div>\n",
       "\n",
       "            <div style=\"float: left; left: -5.5em; top: -4.42em; position: relative;  font-weight: bold;\">AutoML's prediction</div>\n",
       "            <div style=\"float: left; left: -8.22em; top: -2.9em; position: relative;  font-weight: bold;\">0.696</div>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "exp = lime.explain_instance(test.loc[18396], labels=(0, 1), perturb_column='comment_text', n_features=20) \n",
    "exp.visualize_in_notebook(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding some happy words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "test.loc[18396] = \"Okay , thanks . I will do so . happy dumbass please\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "test: 100%|██████████| 157/157 [00:01<00:00, 86.02it/s]\n",
      "test: 100%|██████████| 157/157 [00:01<00:00, 87.59it/s]\n",
      "test: 100%|██████████| 157/157 [00:01<00:00, 85.69it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #FFF4F4;\">Okay</span> <span style=\"background-color: #FFFEFE;\"> , </span> <span style=\"background-color: #FFECEC;\">thanks</span> <span style=\"background-color: #FFFEFE;\"> . </span> <span style=\"background-color: #FFFEFE;\">I</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FCFCFF;\">will</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFEFE;\">do</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFCFC;\">so</span> <span style=\"background-color: #FFFEFE;\"> . </span> <span style=\"background-color: #FFEEEE;\">happy</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #8080FF;\">dumbass</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFF8F8;\">please</span></p>\n",
       "        </div>\n",
       "        \n",
       "        <p style=\"text-align: center\">\n",
       "            Class mapping\n",
       "        </p>\n",
       "        <div style=\"margin-left: 10%; margin-right: 10%; \">\n",
       "            <div id=\"grad\" style=\"background: linear-gradient(90deg, #FF7E7E 0%, #FFFEFE 50%, #8080FF 100%); border: 1px solid black; margin-left: 5.0em; margin-right: 5.0em; float: center; \">\n",
       "                <p style=\"text-align:left; margin-left: 1%; margin-right: 1%; color: white;\">\n",
       "                    Class: 0\n",
       "                    <span style=\"float:right;\">\n",
       "                        Class: 1\n",
       "                    </span>\n",
       "                </p>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"margin-left: 5.0em; margin-right: 5.0em; \">\n",
       "                <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-7.0</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-5.6</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-4.2</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-2.8</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-1.4</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.0</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">1.4</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">2.8</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">4.2</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">5.6</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 0%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">7.0</span></div>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"float: right; right: 0.75em; top: -3em; position: relative; font-weight: bold;\">Scale</div>\n",
       "            <div style=\"float: right; right: -2em; top: -2.9em; position: relative; font-weight: bold;\">✕ 1e-01</div>\n",
       "\n",
       "            <div style=\"float: left; left: -5.5em; top: -4.42em; position: relative;  font-weight: bold;\">AutoML's prediction</div>\n",
       "            <div style=\"float: left; left: -8.22em; top: -2.9em; position: relative;  font-weight: bold;\">0.513</div>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "exp = lime.explain_instance(test.loc[18396], labels=(0, 1), perturb_column='comment_text', n_features=20) \n",
    "exp.visualize_in_notebook(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More happy words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "test.loc[18396] = \"Okay , thanks . I will do so . happy cheerful joyfull glorious elated dumbass please\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "test: 100%|██████████| 157/157 [00:02<00:00, 75.00it/s]\n",
      "test: 100%|██████████| 157/157 [00:02<00:00, 74.62it/s]\n",
      "test: 100%|██████████| 157/157 [00:02<00:00, 74.52it/s]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #FFECEC;\">Okay</span> <span style=\"background-color: #FFFEFE;\"> , </span> <span style=\"background-color: #FFDCDC;\">thanks</span> <span style=\"background-color: #FFFEFE;\"> . </span> <span style=\"background-color: #FFFEFE;\">I</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FAFAFF;\">will</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFEFE;\">do</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FEFEFF;\">so</span> <span style=\"background-color: #FFFEFE;\"> . </span> <span style=\"background-color: #FFE6E6;\">happy</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFEAEA;\">cheerful</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFF4F4;\">joyfull</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFCFC;\">glorious</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFFAFA;\">elated</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #8080FF;\">dumbass</span> <span style=\"background-color: #FFFEFE;\"> </span> <span style=\"background-color: #FFF2F2;\">please</span></p>\n",
       "        </div>\n",
       "        \n",
       "        <p style=\"text-align: center\">\n",
       "            Class mapping\n",
       "        </p>\n",
       "        <div style=\"margin-left: 10%; margin-right: 10%; \">\n",
       "            <div id=\"grad\" style=\"background: linear-gradient(90deg, #FF7E7E 0%, #FFFEFE 50%, #8080FF 100%); border: 1px solid black; margin-left: 5.0em; margin-right: 5.0em; float: center; \">\n",
       "                <p style=\"text-align:left; margin-left: 1%; margin-right: 1%; color: white;\">\n",
       "                    Class: 0\n",
       "                    <span style=\"float:right;\">\n",
       "                        Class: 1\n",
       "                    </span>\n",
       "                </p>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"margin-left: 5.0em; margin-right: 5.0em; \">\n",
       "                <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-4.9</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-3.9</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-2.9</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-2.0</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -1.085em\">-1.0</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">0.0</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">1.0</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">2.0</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">2.9</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 9.97%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">3.9</span></div> <div style=\"border-left: 1px solid black; height: 18px; float: left; width: 0%;\"><span style=\"position: relative; top: 1em; left: -0.7em\">4.9</span></div>\n",
       "            </div>\n",
       "\n",
       "            <div style=\"float: right; right: 0.75em; top: -3em; position: relative; font-weight: bold;\">Scale</div>\n",
       "            <div style=\"float: right; right: -2em; top: -2.9em; position: relative; font-weight: bold;\">✕ 1e-01</div>\n",
       "\n",
       "            <div style=\"float: left; left: -5.5em; top: -4.42em; position: relative;  font-weight: bold;\">AutoML's prediction</div>\n",
       "            <div style=\"float: left; left: -8.22em; top: -2.9em; position: relative;  font-weight: bold;\">0.175</div>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "exp = lime.explain_instance(test.loc[18396], labels=(0, 1), perturb_column='comment_text', n_features=20) \n",
    "exp.visualize_in_notebook(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## L2X for Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this part the [BeerAdvocate](http://people.csail.mit.edu/taolei/beer/) we will use. The dataset contains the reviews on alcoholic drinks (texutal comment + 5 attributes: overview, taste, plate, aroma, appearance). For this experiment we will use only ``appearance`` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GET: 200 CODE\n",
      "Started downloading...\n",
      "Downloaded.\n"
     ]
    }
   ],
   "source": [
    "def download_from_gdrive(file_id, file_name, chunk_size=2**15):\n",
    "    import requests\n",
    "    \n",
    "    def handle_warning(res):\n",
    "        for k, v in res.cookies.items():\n",
    "            if k.startswith(\"download_warning\"):\n",
    "                return v\n",
    "            \n",
    "    template_url = \"https://docs.google.com/uc?export=download\"\n",
    "    session = requests.Session()\n",
    "    res = session.get(template_url, params={\"id\": file_id}, stream=True)\n",
    "    print('GET: {} CODE'.format(res.status_code))\n",
    "    token = handle_warning(res)\n",
    "    if token:\n",
    "        res = session.get(template_url, params={\"id\": file_id, \"confirm\": token}, stream=True)\n",
    "    print('Started downloading...')\n",
    "    with open(file_name, 'wb') as f:\n",
    "        for chunk in res.iter_content(chunk_size):\n",
    "            if chunk:\n",
    "                f.write(chunk)\n",
    "    print('Downloaded.')\n",
    "\n",
    "    \n",
    "download_from_gdrive('1s8PG13Y0BvYM67nNL0EQpdgB5S4gJK9r', 'beeradvocate.tar.gz')\n",
    "shutil.unpack_archive('beeradvocate.tar.gz', '.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Appearance</th>\n",
       "      <th>Aroma</th>\n",
       "      <th>Palate</th>\n",
       "      <th>Taste</th>\n",
       "      <th>Overall</th>\n",
       "      <th>Review</th>\n",
       "      <th>tokens_number</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.5</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2.5</td>\n",
       "      <td>1.5</td>\n",
       "      <td>1.5</td>\n",
       "      <td>the main problem with this beer is that it has...</td>\n",
       "      <td>62</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>it is very unfortunate this situation we have ...</td>\n",
       "      <td>179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.0</td>\n",
       "      <td>2.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>2.0</td>\n",
       "      <td>appearance is a light golden yellow with a thi...</td>\n",
       "      <td>79</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4.5</td>\n",
       "      <td>3.5</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.5</td>\n",
       "      <td>3.0</td>\n",
       "      <td>it has a great color to the body . this beer p...</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4.0</td>\n",
       "      <td>4.5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.5</td>\n",
       "      <td>1.0</td>\n",
       "      <td>though this beer is , or course , not carbonat...</td>\n",
       "      <td>246</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Appearance  Aroma  Palate  Taste  Overall  \\\n",
       "0         1.5    1.5     2.5    1.5      1.5   \n",
       "1         2.0    2.0     3.0    2.0      3.0   \n",
       "2         4.0    2.5     3.0    1.5      2.0   \n",
       "3         4.5    3.5     2.0    3.5      3.0   \n",
       "4         4.0    4.5     1.0    1.5      1.0   \n",
       "\n",
       "                                              Review  tokens_number  \n",
       "0  the main problem with this beer is that it has...             62  \n",
       "1  it is very unfortunate this situation we have ...            179  \n",
       "2  appearance is a light golden yellow with a thi...             79  \n",
       "3  it has a great color to the body . this beer p...             87  \n",
       "4  though this beer is , or course , not carbonat...            246  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data = pd.read_csv('./datasets/reviews.aspect0.train.csv')\n",
    "valid_data = pd.read_csv('./datasets/reviews.aspect0.heldout.csv')\n",
    "\n",
    "train_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train AutoML\n",
    "\n",
    "In this part we use BERT-Base model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[12:38:00] Stdout logging level is INFO2.\n",
      "[12:38:00] Task: reg\n",
      "\n",
      "[12:38:00] Start automl preset with listed constraints:\n",
      "[12:38:00] - time: 3600.00 seconds\n",
      "[12:38:00] - CPU: 1 cores\n",
      "[12:38:00] - memory: 16 GB\n",
      "\n",
      "[12:38:00] \u001b[1mTrain data shape: (80000, 7)\u001b[0m\n",
      "\n",
      "[12:38:01] Layer \u001b[1m1\u001b[0m train process start. Time left 3599.63 secs\n",
      "[12:38:01] Start fitting \u001b[1mLvl_0_Pipe_0_Mod_0_TorchNN\u001b[0m ...\n",
      "[12:38:01] ===== Start working with \u001b[1mfold 0\u001b[0m for \u001b[1mLvl_0_Pipe_0_Mod_0_TorchNN\u001b[0m =====\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train (loss=0.755747): 100%|██████████| 1667/1667 [06:45<00:00,  4.11it/s]\n",
      "val: 100%|██████████| 834/834 [02:04<00:00,  6.68it/s]\n",
      "train (loss=0.442306): 100%|██████████| 1667/1667 [06:48<00:00,  4.08it/s]\n",
      "val: 100%|██████████| 834/834 [02:05<00:00,  6.66it/s]\n",
      "train (loss=0.344638): 100%|██████████| 1667/1667 [06:52<00:00,  4.04it/s]\n",
      "val: 100%|██████████| 834/834 [02:06<00:00,  6.61it/s]\n",
      "val: 100%|██████████| 834/834 [02:05<00:00,  6.64it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[13:07:23] ===== Start working with \u001b[1mfold 1\u001b[0m for \u001b[1mLvl_0_Pipe_0_Mod_0_TorchNN\u001b[0m =====\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train (loss=0.760973): 100%|██████████| 1667/1667 [06:51<00:00,  4.05it/s]\n",
      "val: 100%|██████████| 834/834 [02:06<00:00,  6.62it/s]\n",
      "train (loss=0.44357): 100%|██████████| 1667/1667 [06:50<00:00,  4.06it/s] \n",
      "val: 100%|██████████| 834/834 [02:06<00:00,  6.61it/s]\n",
      "train (loss=0.343338): 100%|██████████| 1667/1667 [06:49<00:00,  4.07it/s]\n",
      "val: 100%|██████████| 834/834 [02:05<00:00,  6.66it/s]\n",
      "val: 100%|██████████| 834/834 [02:05<00:00,  6.66it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[13:36:29] Time limit exceeded after calculating fold 1\n",
      "\n",
      "[13:36:29] Fitting \u001b[1mLvl_0_Pipe_0_Mod_0_TorchNN\u001b[0m finished. score = \u001b[1m-0.46728458911890136\u001b[0m\n",
      "[13:36:29] \u001b[1mLvl_0_Pipe_0_Mod_0_TorchNN\u001b[0m fitting and predicting completed\n",
      "[13:36:29] Time left 91.29 secs\n",
      "\n",
      "[13:36:29] Time limit exceeded in one of the tasks. AutoML will blend level 1 models.\n",
      "\n",
      "[13:36:29] \u001b[1mLayer 1 training completed.\u001b[0m\n",
      "\n",
      "[13:36:29] \u001b[1mAutoml preset training completed in 3508.71 seconds\u001b[0m\n",
      "\n",
      "[13:36:29] Model description:\n",
      "Final prediction for new objects (level 0) = \n",
      "\t 1.00000 * (2 averaged models Lvl_0_Pipe_0_Mod_0_TorchNN) \n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "test: 100%|██████████| 313/313 [00:47<00:00,  6.63it/s]\n",
      "test: 100%|██████████| 313/313 [00:47<00:00,  6.64it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Check scores:\n",
      "OOF score: 0.46728458911890136\n",
      "TEST score: 0.43322843977913716\n"
     ]
    }
   ],
   "source": [
    "roles = {\n",
    "    'text': ['Review'], \n",
    "    'drop': ['tokens_number', 'Aroma', 'Palete', 'Taste', 'Overall'],\n",
    "    'target': 'Appearance'\n",
    "}\n",
    "\n",
    "task = Task('reg')\n",
    "\n",
    "automl = TabularNLPAutoML(\n",
    "    task=task,\n",
    "    timeout=3600,\n",
    "    cpu_limit=1,\n",
    "    gpu_ids='1',\n",
    "    general_params={\n",
    "        'nested_cv': False,\n",
    "        'use_algos': [['nn']],\n",
    "        'n_folds': 3\n",
    "    },\n",
    "    reader_params={\n",
    "        'cv': 3\n",
    "    },\n",
    "    autonlp_params={\n",
    "        'sent_scaler': 'l2'\n",
    "    },\n",
    "    text_params={\n",
    "        'lang': 'en',\n",
    "        'bert_model': 'bert-base-uncased'\n",
    "    },\n",
    "    nn_params={\n",
    "        'opt_params': {'lr': 1e-5},\n",
    "        'max_length': 128,\n",
    "        'bs': 32,\n",
    "        'n_epochs': 7,\n",
    "    },\n",
    ")\n",
    "\n",
    "oof_pred = automl.fit_predict(train_data, roles=roles, verbose=2)\n",
    "test_pred = automl.predict(valid_data)\n",
    "not_nan = np.any(~np.isnan(oof_pred.data), axis=1)\n",
    "\n",
    "print('Check scores:')\n",
    "print('OOF score: {}'.format(mean_squared_error(train_data[roles['target']].values[not_nan], oof_pred.data[not_nan][:, 0])))\n",
    "print('TEST score: {}'.format(mean_squared_error(valid_data[roles['target']].values, test_pred.data[:, 0])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# >>> about 2gb\n",
    "with open('apperance_model.pkl', 'wb') as f:\n",
    "    pickle.dump(automl, f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[13:38:29] Stdout logging level is INFO2.\n"
     ]
    }
   ],
   "source": [
    "with open('apperance_model.pkl', 'rb') as f:\n",
    "    automl = pickle.load(f)\n",
    "automl.set_verbosity_level(2)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## L2X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Algorithm.\n",
    "\n",
    "0. The general idea of method is find the most informative subset of tokens with respect to target using [Mutual Information](https://en.wikipedia.org/wiki/Mutual_information). The number of tokens in this subset is fixed and equals ```n_important```.\n",
    "\n",
    "1. There is may be some misunderstanding with tokenization that used inside models in automl and tokenization in this method. L2X has its own tokenization, so they are different. If it isn't set we infer it from default tokenization for language in ``text_params`` of ```TabularNLPAutoML```. Else you can set it with language: ``'ru'`` or ``'en'`` for russian and english languages, respectively. Also it can be scepcified as callable function that from string produces list of tokens.\n",
    "\n",
    "2. After tokenization sentence was presented as the matrix of embedding vectors (you can specify ``embedder``  or randomly initialized embeddings will be used). Not important vectors of this matrix will be masked (important tokens selected with Token Importance + Subset Sampler blocks), and the other use for model (Distil model), that tries to imitate the original automl model (learns to predict the same outputs).\n",
    "\n",
    "3. Scheme of L2X:\n",
    "\n",
    "\n",
    "<img src=\"../../imgs/GENERALL2X2.jpg\" alt=\"L2X scheme\" style=\"width:100%;\"/>\n",
    "\n",
    "4. Some info about parameters:\n",
    " - ``n_important`` - number of important tokens;\n",
    " - ``temperature`` - initial temperature used in gumbel softmax trick;\n",
    " - ``train_device`` - device used for training;\n",
    " - ``inference_device`` - device used for inference;\n",
    " - ``verbose`` - verbose mode;\n",
    " - ``binning_mode`` - for training we use batch sampling by the length of sequence. So, batch formed only by the sequences from the respect bin. This parameter used for method of automatic binning border choosing. There are two of them: ``'linear'`` (min-max binning, like linspace), ``'hist'`` (histogram binning).\n",
    " - ``bins_number`` - number of bins in batch sampling process;\n",
    " - ``n_epochs`` - number of epochs of training of the L2X;\n",
    " - ``learning_rate`` - learning rate of L2X model;\n",
    " - ``patience`` - number of epoches before learning rate decreasing (``torch.optim.lr_scheduler.ReduceLROnPlateu``);\n",
    " - ``extreme_patience`` - number of epoches before early stopping by the validation dataset;\n",
    " - ``train_batch_size`` - size of batch for training process;\n",
    " - ``valid_batch_size`` - size of batch for validation process;\n",
    " - ``temp_anneal_factor`` - annealing factor for temperature. The temperature will be multiplied by this coefficient every epoch.\n",
    " - ``importance_sampler`` - specifies method of sampling importance (there are two of them ``'gumbeltopk'`` - method from the original paper, ``'softsub'`` - another method);\n",
    " - `max_vocab_length` - maximum length of vocabular (vocabular build up from ``max_vocab_length`` the most frequent tokens). If ``max_vocab_length`` is ``-1`` then include all in train set.\n",
    " - ``embedder`` - embedding dictionary or path to fasttext/dict of embeddings.\n",
    " \n",
    "5. Some links for more info about L2X:\n",
    "    1. [Gumbel Softmax Trick](https://arxiv.org/abs/1611.00712)\n",
    "    2. [L2X Original Paper](https://arxiv.org/abs/1802.07814)\n",
    "    3. [SoftSub Paper](https://arxiv.org/abs/1901.10517)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "test: 100%|██████████| 2500/2500 [06:14<00:00,  6.67it/s]\n",
      "test: 100%|██████████| 2500/2500 [06:15<00:00,  6.66it/s]\n",
      "test: 100%|██████████| 313/313 [00:47<00:00,  6.66it/s]\n",
      "test: 100%|██████████| 313/313 [00:46<00:00,  6.66it/s]\n",
      "train nll (loss=7.8830): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.12it/s]\n",
      "train nll (loss=1.4016): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.63it/s]\n",
      "train nll (loss=1.3859): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.53it/s]\n",
      "train nll (loss=1.3684): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.57it/s]\n",
      "train nll (loss=1.0265): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.44it/s]\n",
      "train nll (loss=0.7086): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.46it/s]\n",
      "train nll (loss=0.6344): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.62it/s]\n",
      "train nll (loss=0.5779): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.68it/s]\n",
      "train nll (loss=0.5318): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.22it/s]\n",
      "train nll (loss=0.4962): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.56it/s]\n",
      "train nll (loss=0.4575): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.69it/s]\n",
      "train nll (loss=0.4233): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.65it/s]\n",
      "train nll (loss=0.3882): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.58it/s]\n",
      "train nll (loss=0.3574): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.59it/s]\n",
      "train nll (loss=0.3326): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.41it/s]\n",
      "train nll (loss=0.3177): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.70it/s]\n",
      "train nll (loss=0.2997): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.50it/s]\n",
      "train nll (loss=0.2885): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.50it/s]\n",
      "train nll (loss=0.2768): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.24it/s]\n",
      "train nll (loss=0.2667): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.41it/s]\n",
      "train nll (loss=0.2569): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.40it/s]\n",
      "train nll (loss=0.2500): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.54it/s]\n",
      "train nll (loss=0.2439): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.26it/s]\n",
      "train nll (loss=0.2349): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.54it/s]\n",
      "train nll (loss=0.2278): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.25it/s]\n",
      "train nll (loss=0.2244): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.35it/s]\n",
      "train nll (loss=0.2220): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.35it/s]\n",
      "train nll (loss=0.2158): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.36it/s]\n",
      "train nll (loss=0.2110): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.45it/s]\n",
      "train nll (loss=0.2080): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.43it/s]\n",
      "train nll (loss=0.2050): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.40it/s]\n",
      "train nll (loss=0.2003): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.38it/s]\n",
      "train nll (loss=0.1977): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.43it/s]\n",
      "train nll (loss=0.1925): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.42it/s]\n",
      "train nll (loss=0.1919): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.55it/s]\n",
      "train nll (loss=0.1888): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.49it/s]\n",
      "train nll (loss=0.1842): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.37it/s]\n",
      "train nll (loss=0.1841): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.44it/s]\n",
      "train nll (loss=0.1820): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.52it/s]\n",
      "train nll (loss=0.1777): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.37it/s]\n",
      "train nll (loss=0.1785): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.27it/s]\n",
      "train nll (loss=0.1778): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.40it/s]\n",
      "train nll (loss=0.1748): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.40it/s]\n",
      "train nll (loss=0.1719): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.42it/s]\n",
      "train nll (loss=0.1704): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.58it/s]\n",
      "train nll (loss=0.1715): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.74it/s]\n",
      "train nll (loss=0.1715): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.48it/s]\n",
      "train nll (loss=0.1734): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.68it/s]\n",
      "train nll (loss=0.1732): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.65it/s]\n",
      "train nll (loss=0.1781): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.58it/s]\n",
      "train nll (loss=0.1770): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.57it/s]\n",
      "train nll (loss=0.1737): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.66it/s]\n",
      "train nll (loss=0.1728): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.59it/s]\n",
      "train nll (loss=0.1731): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.48it/s]\n",
      "train nll (loss=0.1708): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.26it/s]\n",
      "train nll (loss=0.1696): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.35it/s]\n",
      "train nll (loss=0.1699): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.01it/s]\n",
      "train nll (loss=0.1699): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.39it/s]\n",
      "train nll (loss=0.1681): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.37it/s]\n",
      "train nll (loss=0.1682): 100%|█████████▉| 1249/1251 [00:45<00:00, 27.39it/s]\n",
      "train nll (loss=0.1684): 100%|█████████▉| 1249/1251 [00:43<00:00, 28.57it/s]\n",
      "train nll (loss=0.1666): 100%|█████████▉| 1249/1251 [00:46<00:00, 26.91it/s]\n",
      "train nll (loss=0.1659): 100%|█████████▉| 1249/1251 [00:51<00:00, 24.20it/s]\n",
      "train nll (loss=0.1656): 100%|█████████▉| 1249/1251 [00:46<00:00, 27.09it/s]\n",
      "train nll (loss=0.1665): 100%|█████████▉| 1249/1251 [00:43<00:00, 28.92it/s]\n",
      "train nll (loss=0.1676): 100%|█████████▉| 1249/1251 [00:43<00:00, 28.65it/s]\n",
      "train nll (loss=0.1657): 100%|█████████▉| 1249/1251 [00:43<00:00, 29.02it/s]\n",
      "train nll (loss=0.1651): 100%|█████████▉| 1249/1251 [00:43<00:00, 28.92it/s]\n",
      "train nll (loss=0.1631): 100%|█████████▉| 1249/1251 [00:43<00:00, 28.87it/s]\n",
      "train nll (loss=0.1634): 100%|█████████▉| 1249/1251 [00:43<00:00, 28.79it/s]\n",
      "train nll (loss=0.1634): 100%|█████████▉| 1249/1251 [00:43<00:00, 28.77it/s]\n",
      "train nll (loss=0.1626): 100%|█████████▉| 1249/1251 [00:43<00:00, 28.40it/s]\n",
      "train nll (loss=0.1631): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.35it/s]\n",
      "train nll (loss=0.1613): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.44it/s]\n",
      "train nll (loss=0.1614): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.63it/s]\n",
      "train nll (loss=0.1638): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.51it/s]\n",
      "train nll (loss=0.1633): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.52it/s]\n",
      "train nll (loss=0.1618): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.39it/s]\n",
      "train nll (loss=0.1612): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.43it/s]\n",
      "train nll (loss=0.1628): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.34it/s]\n",
      "train nll (loss=0.1616): 100%|█████████▉| 1249/1251 [00:41<00:00, 29.99it/s]\n",
      "train nll (loss=0.1618): 100%|█████████▉| 1249/1251 [00:41<00:00, 29.77it/s]\n",
      "train nll (loss=0.1594): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.29it/s]\n",
      "train nll (loss=0.1617): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.55it/s]\n",
      "train nll (loss=0.1617): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.27it/s]\n",
      "train nll (loss=0.1610): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.35it/s]\n",
      "train nll (loss=0.1590): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.35it/s]\n",
      "train nll (loss=0.1602): 100%|█████████▉| 1249/1251 [00:43<00:00, 28.49it/s]\n",
      "train nll (loss=0.1602): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.07it/s]\n",
      "train nll (loss=0.1613): 100%|█████████▉| 1249/1251 [00:43<00:00, 28.80it/s]\n",
      "train nll (loss=0.1620): 100%|█████████▉| 1249/1251 [00:43<00:00, 28.86it/s]\n",
      "train nll (loss=0.1593): 100%|█████████▉| 1249/1251 [00:43<00:00, 28.95it/s]\n",
      "train nll (loss=0.1612): 100%|█████████▉| 1249/1251 [00:43<00:00, 28.89it/s]\n",
      "train nll (loss=0.1620): 100%|█████████▉| 1249/1251 [00:43<00:00, 28.93it/s]\n",
      "train nll (loss=0.1614): 100%|█████████▉| 1249/1251 [00:44<00:00, 28.26it/s]\n",
      "train nll (loss=0.1630): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.19it/s]\n",
      "train nll (loss=0.1665): 100%|█████████▉| 1249/1251 [00:43<00:00, 28.75it/s]\n",
      "train nll (loss=0.1605): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.48it/s]\n",
      "train nll (loss=0.1605): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.01it/s]\n",
      "train nll (loss=0.1636): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.58it/s]\n",
      "train nll (loss=0.1617): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.08it/s]\n",
      "train nll (loss=0.1635): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.44it/s]\n",
      "train nll (loss=0.1606): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.65it/s]\n",
      "train nll (loss=0.1631): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.45it/s]\n",
      "train nll (loss=0.1645): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.60it/s]\n",
      "train nll (loss=0.1652): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.37it/s]\n",
      "train nll (loss=0.1641): 100%|█████████▉| 1249/1251 [00:41<00:00, 29.96it/s]\n",
      "train nll (loss=0.1669): 100%|█████████▉| 1249/1251 [00:43<00:00, 28.80it/s]\n",
      "train nll (loss=0.1610): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.11it/s]\n",
      "train nll (loss=0.1630): 100%|█████████▉| 1249/1251 [00:43<00:00, 28.96it/s]\n",
      "train nll (loss=0.1644): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.11it/s]\n",
      "train nll (loss=0.1681): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.06it/s]\n",
      "train nll (loss=0.1691): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.12it/s]\n",
      "train nll (loss=0.1728): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.15it/s]\n",
      "train nll (loss=0.1710): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.16it/s]\n",
      "train nll (loss=0.1679): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.30it/s]\n",
      "train nll (loss=0.1697): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.31it/s]\n",
      "train nll (loss=0.1669): 100%|█████████▉| 1249/1251 [00:41<00:00, 29.92it/s]\n",
      "train nll (loss=0.1708): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.28it/s]\n",
      "train nll (loss=0.1662): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.56it/s]\n",
      "train nll (loss=0.1739): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.34it/s]\n",
      "train nll (loss=0.1977): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.68it/s]\n",
      "train nll (loss=0.1844): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.72it/s]\n",
      "train nll (loss=0.1685): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.17it/s]\n",
      "train nll (loss=0.1664): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.48it/s]\n",
      "train nll (loss=0.1760): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.63it/s]\n",
      "train nll (loss=0.1678): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.71it/s]\n",
      "train nll (loss=0.1681): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.25it/s]\n",
      "train nll (loss=0.1806): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.67it/s]\n",
      "train nll (loss=0.1724): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.70it/s]\n",
      "train nll (loss=0.1691): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.67it/s]\n",
      "train nll (loss=0.1712): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.71it/s]\n",
      "train nll (loss=0.1762): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.55it/s]\n",
      "train nll (loss=0.1655): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.00it/s]\n",
      "train nll (loss=0.1860): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.72it/s]\n",
      "train nll (loss=0.1897): 100%|█████████▉| 1249/1251 [00:41<00:00, 29.76it/s]\n",
      "train nll (loss=0.1770): 100%|█████████▉| 1249/1251 [00:41<00:00, 30.45it/s]\n",
      "train nll (loss=0.1796): 100%|█████████▉| 1249/1251 [00:42<00:00, 29.65it/s]\n",
      "train nll (loss=0.1816): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.48it/s]\n",
      "train nll (loss=0.1870): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.75it/s]\n",
      "train nll (loss=0.1797): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.57it/s]\n",
      "train nll (loss=0.1878): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.75it/s]\n",
      "train nll (loss=0.1903): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.72it/s]\n",
      "train nll (loss=0.1787): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.63it/s]\n",
      "train nll (loss=0.1815): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.70it/s]\n",
      "train nll (loss=0.1781): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.68it/s]\n",
      "train nll (loss=0.1749): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.68it/s]\n",
      "train nll (loss=0.1735): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.51it/s]\n",
      "train nll (loss=0.1675): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.56it/s]\n",
      "train nll (loss=0.1671): 100%|█████████▉| 1249/1251 [00:40<00:00, 30.58it/s]\n"
     ]
    }
   ],
   "source": [
    "l2x = L2XTextExplainer(automl, train_device='cuda:1',\n",
    "                       inference_device='cuda:1',\n",
    "                       embedding_dim=300,\n",
    "                       gamma=0.1, temperature=2, temp_anneal_factor=0.95,\n",
    "                       n_epochs=200, importance_sampler='gumbeltopk',\n",
    "                       n_important=20, patience=25,\n",
    "                       extreme_patience=30, trainable_embeds=True)\n",
    "\n",
    "l2x.fit(train_data, valid_data, cols_to_explain='Review')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "expl_train = l2x['Review'].explain_instances(train_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "expl_valid = l2x['Review'].explain_instances(valid_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #FFFFFF;\">&lt;START&gt;</span> <span style=\"background-color: #FFFFFF;\">lot</span> <span style=\"background-color: #6DB5F5;\">exploder</span> <span style=\"background-color: #6DB5F5;\">lost</span> <span style=\"background-color: #FFFFFF;\">about</span> <span style=\"background-color: #FFFFFF;\">3</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">beer</span> <span style=\"background-color: #FFFFFF;\">down</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">drain</span> <span style=\"background-color: #FFFFFF;\">as</span> <span style=\"background-color: #FFFFFF;\">foam</span> <span style=\"background-color: #6DB5F5;\">whats</span> <span style=\"background-color: #6DB5F5;\">left</span> <span style=\"background-color: #6DB5F5;\">is</span> <span style=\"background-color: #6DB5F5;\">a</span> <span style=\"background-color: #6DB5F5;\">cloudy</span> <span style=\"background-color: #6DB5F5;\">medium</span> <span style=\"background-color: #6DB5F5;\">brown</span> <span style=\"background-color: #6DB5F5;\">color</span> <span style=\"background-color: #6DB5F5;\">with</span> <span style=\"background-color: #6DB5F5;\">floaties</span> <span style=\"background-color: #6DB5F5;\">plenty</span> <span style=\"background-color: #6DB5F5;\">of</span> <span style=\"background-color: #6DB5F5;\">head</span> <span style=\"background-color: #6DB5F5;\">obviously</span> <span style=\"background-color: #6DB5F5;\">which</span> <span style=\"background-color: #6DB5F5;\">dissipates</span> <span style=\"background-color: #6DB5F5;\">quickly</span> <span style=\"background-color: #6DB5F5;\">aroma</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">tons</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">malt</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">dark</span> <span style=\"background-color: #FFFFFF;\">fruit</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">flavor</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">again</span> <span style=\"background-color: #FFFFFF;\">very</span> <span style=\"background-color: #FFFFFF;\">fruity</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">bready</span> <span style=\"background-color: #FFFFFF;\">malt</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">caramel</span> <span style=\"background-color: #FFFFFF;\">notes</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">bit</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">roast</span> <span style=\"background-color: #FFFFFF;\">malt</span> <span style=\"background-color: #FFFFFF;\">no</span> <span style=\"background-color: #FFFFFF;\">hint</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">spices</span> <span style=\"background-color: #FFFFFF;\">anywhere</span> <span style=\"background-color: #FFFFFF;\">full</span> <span style=\"background-color: #FFFFFF;\">bodied</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">plenty</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">silky</span> <span style=\"background-color: #FFFFFF;\">crispness</span> <span style=\"background-color: #FFFFFF;\"></span> <span style=\"background-color: #FFFFFF;\">&lt;PAD&gt;</span></p>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "expl_valid[66].visualize_in_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #FFFFFF;\">&lt;START&gt;</span> <span style=\"background-color: #FFFFFF;\">whoa</span> <span style=\"background-color: #6DB5F5;\">is</span> <span style=\"background-color: #FFFFFF;\">right</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">one</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">big</span> <span style=\"background-color: #FFFFFF;\">brew</span> <span style=\"background-color: #FFFFFF;\">in</span> <span style=\"background-color: #FFFFFF;\">my</span> <span style=\"background-color: #FFFFFF;\">opinion</span> <span style=\"background-color: #FFFFFF;\">hence</span> <span style=\"background-color: #FFFFFF;\">its</span> <span style=\"background-color: #FFFFFF;\">name</span> <span style=\"background-color: #6DB5F5;\">pours</span> <span style=\"background-color: #6DB5F5;\">a</span> <span style=\"background-color: #6DB5F5;\">thick</span> <span style=\"background-color: #6DB5F5;\">creamy</span> <span style=\"background-color: #6DB5F5;\">head</span> <span style=\"background-color: #6DB5F5;\">and</span> <span style=\"background-color: #6DB5F5;\">has</span> <span style=\"background-color: #6DB5F5;\">a</span> <span style=\"background-color: #6DB5F5;\">dark</span> <span style=\"background-color: #6DB5F5;\">brown</span> <span style=\"background-color: #6DB5F5;\">color</span> <span style=\"background-color: #6DB5F5;\">with</span> <span style=\"background-color: #6DB5F5;\">hints</span> <span style=\"background-color: #6DB5F5;\">of</span> <span style=\"background-color: #6DB5F5;\">amber</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">taste</span> <span style=\"background-color: #FFFFFF;\">ha</span> <span style=\"background-color: #FFFFFF;\">thick</span> <span style=\"background-color: #FFFFFF;\">hops</span> <span style=\"background-color: #FFFFFF;\">in</span> <span style=\"background-color: #FFFFFF;\">here</span> <span style=\"background-color: #FFFFFF;\">think</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">biting</span> <span style=\"background-color: #FFFFFF;\">into</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">big</span> <span style=\"background-color: #FFFFFF;\">juicy</span> <span style=\"background-color: #FFFFFF;\">fruit</span> <span style=\"background-color: #FFFFFF;\">terrapin</span> <span style=\"background-color: #FFFFFF;\">comes</span> <span style=\"background-color: #FFFFFF;\">out</span> <span style=\"background-color: #FFFFFF;\">strong</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">seasonal</span> <span style=\"background-color: #FFFFFF;\">taste</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">alcohol</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">well</span> <span style=\"background-color: #FFFFFF;\">hidden</span> <span style=\"background-color: #FFFFFF;\">but</span> <span style=\"background-color: #FFFFFF;\">will</span> <span style=\"background-color: #FFFFFF;\">creap</span> <span style=\"background-color: #FFFFFF;\">up</span> <span style=\"background-color: #FFFFFF;\">on</span> <span style=\"background-color: #FFFFFF;\">you</span> <span style=\"background-color: #FFFFFF;\">in</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">hurry</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">have</span> <span style=\"background-color: #FFFFFF;\">found</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">most</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">year</span> <span style=\"background-color: #FFFFFF;\">for</span> <span style=\"background-color: #FFFFFF;\">some</span> <span style=\"background-color: #FFFFFF;\">reason</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">guess</span> <span style=\"background-color: #FFFFFF;\">they</span> <span style=\"background-color: #FFFFFF;\">distributed</span> <span style=\"background-color: #FFFFFF;\">alot</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">in</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #6DB5F5;\">atlanta</span> <span style=\"background-color: #6DB5F5;\">area</span> <span style=\"background-color: #6DB5F5;\"></span> <span style=\"background-color: #6DB5F5;\">&lt;PAD&gt;</span></p>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "expl_valid[55].visualize_in_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #FFFFFF;\">&lt;START&gt;</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">beer</span> <span style=\"background-color: #6DB5F5;\">pours</span> <span style=\"background-color: #6DB5F5;\">an</span> <span style=\"background-color: #6DB5F5;\">opaque</span> <span style=\"background-color: #6DB5F5;\">light</span> <span style=\"background-color: #6DB5F5;\">copper</span> <span style=\"background-color: #6DB5F5;\">capped</span> <span style=\"background-color: #6DB5F5;\">by</span> <span style=\"background-color: #6DB5F5;\">a</span> <span style=\"background-color: #6DB5F5;\">minimal</span> <span style=\"background-color: #6DB5F5;\">off</span> <span style=\"background-color: #6DB5F5;\">white</span> <span style=\"background-color: #6DB5F5;\">head</span> <span style=\"background-color: #6DB5F5;\">there</span> <span style=\"background-color: #6DB5F5;\">s</span> <span style=\"background-color: #6DB5F5;\">very</span> <span style=\"background-color: #6DB5F5;\">little</span> <span style=\"background-color: #6DB5F5;\">retention</span> <span style=\"background-color: #6DB5F5;\">despite</span> <span style=\"background-color: #6DB5F5;\">a</span> <span style=\"background-color: #FFFFFF;\">robust</span> <span style=\"background-color: #FFFFFF;\">pour</span> <span style=\"background-color: #FFFFFF;\">into</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">glass</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">nose</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">simply</span> <span style=\"background-color: #FFFFFF;\">divine</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">feel</span> <span style=\"background-color: #FFFFFF;\">like</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">just</span> <span style=\"background-color: #FFFFFF;\">pulled</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">freshly</span> <span style=\"background-color: #FFFFFF;\">baked</span> <span style=\"background-color: #FFFFFF;\">pumpkin</span> <span style=\"background-color: #FFFFFF;\">pie</span> <span style=\"background-color: #FFFFFF;\">out</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">oven</span> <span style=\"background-color: #FFFFFF;\">aromas</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">gram</span> <span style=\"background-color: #FFFFFF;\">cracker</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">butterscotch</span> <span style=\"background-color: #FFFFFF;\">covered</span> <span style=\"background-color: #FFFFFF;\">shortbread</span> <span style=\"background-color: #FFFFFF;\">mix</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">sweet</span> <span style=\"background-color: #FFFFFF;\">potato</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">canned</span> <span style=\"background-color: #FFFFFF;\">&lt;UNK&gt;</span> <span style=\"background-color: #FFFFFF;\">pumpkin</span> <span style=\"background-color: #FFFFFF;\">puree</span> <span style=\"background-color: #FFFFFF;\">coconut</span> <span style=\"background-color: #FFFFFF;\">cinnamon</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">hint</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">citrus</span> <span style=\"background-color: #FFFFFF;\">add</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">twist</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">exotic</span> <span style=\"background-color: #FFFFFF;\">liquid</span> <span style=\"background-color: #6DB5F5;\">pumpkin</span> <span style=\"background-color: #FFFFFF;\">pie</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">best</span> <span style=\"background-color: #FFFFFF;\">way</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">describe</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">flavour</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">fine</span> <span style=\"background-color: #FFFFFF;\">pte</span> <span style=\"background-color: #FFFFFF;\">sucre</span> <span style=\"background-color: #FFFFFF;\">crust</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">rich</span> <span style=\"background-color: #FFFFFF;\">pumpkin</span> <span style=\"background-color: #FFFFFF;\">filling</span> <span style=\"background-color: #FFFFFF;\">spiced</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">cinnamon</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">allspice</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">really</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">devilishly</span> <span style=\"background-color: #FFFFFF;\">good</span> <span style=\"background-color: #FFFFFF;\">everything</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">found</span> <span style=\"background-color: #FFFFFF;\">lacking</span> <span style=\"background-color: #FFFFFF;\">in</span> <span style=\"background-color: #FFFFFF;\">previous</span> <span style=\"background-color: #FFFFFF;\">pumpkin</span> <span style=\"background-color: #FFFFFF;\">beers</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">makes</span> <span style=\"background-color: #FFFFFF;\">up</span> <span style=\"background-color: #FFFFFF;\">for</span> <span style=\"background-color: #FFFFFF;\">smooth</span> <span style=\"background-color: #FFFFFF;\">creamy</span> <span style=\"background-color: #FFFFFF;\">macadamia</span> <span style=\"background-color: #FFFFFF;\">nuttiness</span> <span style=\"background-color: #FFFFFF;\">adds</span> <span style=\"background-color: #FFFFFF;\">another</span> <span style=\"background-color: #FFFFFF;\">dimension</span> <span style=\"background-color: #FFFFFF;\">hops</span> <span style=\"background-color: #FFFFFF;\">are</span> <span style=\"background-color: #FFFFFF;\">nearly</span> <span style=\"background-color: #FFFFFF;\">&lt;UNK&gt;</span> <span style=\"background-color: #FFFFFF;\">simple</span> <span style=\"background-color: #FFFFFF;\">there</span> <span style=\"background-color: #FFFFFF;\">for</span> <span style=\"background-color: #FFFFFF;\">balance</span> <span style=\"background-color: #FFFFFF;\">similarly</span> <span style=\"background-color: #FFFFFF;\">absent</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">taste</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">alcohol</span> <span style=\"background-color: #FFFFFF;\">despite</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">whopping</span> <span style=\"background-color: #FFFFFF;\">percentage</span> <span style=\"background-color: #FFFFFF;\">only</span> <span style=\"background-color: #FFFFFF;\">int</span> <span style=\"background-color: #FFFFFF;\">he</span> <span style=\"background-color: #FFFFFF;\">very</span> <span style=\"background-color: #FFFFFF;\">finish</span> <span style=\"background-color: #FFFFFF;\">does</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">pop</span> <span style=\"background-color: #FFFFFF;\">up</span> <span style=\"background-color: #FFFFFF;\">like</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">&lt;UNK&gt;</span> <span style=\"background-color: #FFFFFF;\">child</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">medium</span> <span style=\"background-color: #FFFFFF;\">body</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">medium</span> <span style=\"background-color: #FFFFFF;\">low</span> <span style=\"background-color: #FFFFFF;\">level</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">carbonation</span> <span style=\"background-color: #FFFFFF;\">make</span> <span style=\"background-color: #FFFFFF;\">for</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">surprisingly</span> <span style=\"background-color: #FFFFFF;\">easy</span> <span style=\"background-color: #FFFFFF;\">drinking</span> <span style=\"background-color: #FFFFFF;\">beer</span> <span style=\"background-color: #FFFFFF;\">dangerous</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">best</span> <span style=\"background-color: #FFFFFF;\">pumpkin</span> <span style=\"background-color: #FFFFFF;\">beer</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">ve</span> <span style=\"background-color: #FFFFFF;\">ever</span> <span style=\"background-color: #FFFFFF;\">had</span> <span style=\"background-color: #FFFFFF;\">hands</span> <span style=\"background-color: #FFFFFF;\">down</span> <span style=\"background-color: #FFFFFF;\"></span> <span style=\"background-color: #FFFFFF;\">&lt;PAD&gt;</span></p>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "expl_valid[77].visualize_in_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #FFFFFF;\">&lt;START&gt;</span> <span style=\"background-color: #FFFFFF;\">og</span> <span style=\"background-color: #FFFFFF;\">5</span> <span style=\"background-color: #6DB5F5;\">p</span> <span style=\"background-color: #FFFFFF;\">sg</span> <span style=\"background-color: #FFFFFF;\">046</span> <span style=\"background-color: #FFFFFF;\">1</span> <span style=\"background-color: #FFFFFF;\">abv</span> <span style=\"background-color: #FFFFFF;\">pours</span> <span style=\"background-color: #FFFFFF;\">out</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #6DB5F5;\">clear</span> <span style=\"background-color: #6DB5F5;\">very</span> <span style=\"background-color: #6DB5F5;\">pale</span> <span style=\"background-color: #6DB5F5;\">golden</span> <span style=\"background-color: #6DB5F5;\">forming</span> <span style=\"background-color: #6DB5F5;\">a</span> <span style=\"background-color: #6DB5F5;\">soapy</span> <span style=\"background-color: #6DB5F5;\">white</span> <span style=\"background-color: #6DB5F5;\">head</span> <span style=\"background-color: #6DB5F5;\">with</span> <span style=\"background-color: #6DB5F5;\">decent</span> <span style=\"background-color: #6DB5F5;\">retention</span> <span style=\"background-color: #6DB5F5;\">and</span> <span style=\"background-color: #6DB5F5;\">good</span> <span style=\"background-color: #6DB5F5;\">lacing</span> <span style=\"background-color: #FFFFFF;\">carbonation</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">moderate</span> <span style=\"background-color: #FFFFFF;\">aroma</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">weak</span> <span style=\"background-color: #FFFFFF;\">floral</span> <span style=\"background-color: #FFFFFF;\">hops</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">touch</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">freesia</span> <span style=\"background-color: #FFFFFF;\">corn</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">light</span> <span style=\"background-color: #FFFFFF;\">dryness</span> <span style=\"background-color: #FFFFFF;\">mouthfeel</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">average</span> <span style=\"background-color: #FFFFFF;\">watery</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">light</span> <span style=\"background-color: #FFFFFF;\">body</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">clean</span> <span style=\"background-color: #FFFFFF;\">finish</span> <span style=\"background-color: #FFFFFF;\">taste</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">predominated</span> <span style=\"background-color: #FFFFFF;\">by</span> <span style=\"background-color: #FFFFFF;\">corn</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">hardly</span> <span style=\"background-color: #FFFFFF;\">any</span> <span style=\"background-color: #FFFFFF;\">perceptable</span> <span style=\"background-color: #FFFFFF;\">hop</span> <span style=\"background-color: #FFFFFF;\">flavor</span> <span style=\"background-color: #FFFFFF;\">or</span> <span style=\"background-color: #FFFFFF;\">bitterness</span> <span style=\"background-color: #FFFFFF;\">cleanly</span> <span style=\"background-color: #FFFFFF;\">fermented</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">crisp</span> <span style=\"background-color: #FFFFFF;\">finish</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">very</span> <span style=\"background-color: #FFFFFF;\">simple</span> <span style=\"background-color: #FFFFFF;\">beer</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">only</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">step</span> <span style=\"background-color: #FFFFFF;\">above</span> <span style=\"background-color: #FFFFFF;\">budweiser</span> <span style=\"background-color: #FFFFFF;\">if</span> <span style=\"background-color: #FFFFFF;\">you</span> <span style=\"background-color: #FFFFFF;\">need</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">transition</span> <span style=\"background-color: #FFFFFF;\">someone</span> <span style=\"background-color: #FFFFFF;\">from</span> <span style=\"background-color: #FFFFFF;\">macro</span> <span style=\"background-color: #FFFFFF;\">domestic</span> <span style=\"background-color: #FFFFFF;\">swill</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">an</span> <span style=\"background-color: #FFFFFF;\">average</span> <span style=\"background-color: #FFFFFF;\">pale</span> <span style=\"background-color: #FFFFFF;\">ale</span> <span style=\"background-color: #FFFFFF;\">then</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">would</span> <span style=\"background-color: #FFFFFF;\">be</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">one</span> <span style=\"background-color: #FFFFFF;\">comparable</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">lightest</span> <span style=\"background-color: #FFFFFF;\">tap</span> <span style=\"background-color: #FFFFFF;\">at</span> <span style=\"background-color: #FFFFFF;\">any</span> <span style=\"background-color: #FFFFFF;\">new</span> <span style=\"background-color: #FFFFFF;\">brewpub</span> <span style=\"background-color: #FFFFFF;\">easy</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">drink</span> <span style=\"background-color: #FFFFFF;\">but</span> <span style=\"background-color: #FFFFFF;\">then</span> <span style=\"background-color: #FFFFFF;\">again</span> <span style=\"background-color: #FFFFFF;\">why</span> <span style=\"background-color: #FFFFFF;\">would</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #6DB5F5;\">want</span> <span style=\"background-color: #6DB5F5;\">to</span> <span style=\"background-color: #6DB5F5;\"></span> <span style=\"background-color: #6DB5F5;\">&lt;PAD&gt;</span></p>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "expl_valid[88].visualize_in_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #FFFFFF;\">&lt;START&gt;</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">blend</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">stout</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">bock</span> <span style=\"background-color: #FFFFFF;\">cool</span> <span style=\"background-color: #FFFFFF;\">hopefully</span> <span style=\"background-color: #FFFFFF;\">better</span> <span style=\"background-color: #FFFFFF;\">than</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">blend</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">wheat</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">&lt;UNK&gt;</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">inside</span> <span style=\"background-color: #FFFFFF;\">joke</span> <span style=\"background-color: #FFFFFF;\">for</span> <span style=\"background-color: #FFFFFF;\">anyone</span> <span style=\"background-color: #FFFFFF;\">who</span> <span style=\"background-color: #FFFFFF;\">has</span> <span style=\"background-color: #FFFFFF;\">worked</span> <span style=\"background-color: #FFFFFF;\">as</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">grain</span> <span style=\"background-color: #FFFFFF;\">handler</span> <span style=\"background-color: #FFFFFF;\">not</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">good</span> <span style=\"background-color: #FFFFFF;\">thing</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">know</span> <span style=\"background-color: #FFFFFF;\">some</span> <span style=\"background-color: #FFFFFF;\">beer</span> <span style=\"background-color: #FFFFFF;\">gods</span> <span style=\"background-color: #FFFFFF;\">frown</span> <span style=\"background-color: #FFFFFF;\">on</span> <span style=\"background-color: #6DB5F5;\">the</span> <span style=\"background-color: #6DB5F5;\">whole</span> <span style=\"background-color: #6DB5F5;\">black</span> <span style=\"background-color: #6DB5F5;\">tan</span> <span style=\"background-color: #6DB5F5;\">thing</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">my</span> <span style=\"background-color: #FFFFFF;\">first</span> <span style=\"background-color: #FFFFFF;\">so</span> <span style=\"background-color: #FFFFFF;\">in</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">dive</span> <span style=\"background-color: #FFFFFF;\">in</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">my</span> <span style=\"background-color: #FFFFFF;\">usual</span> <span style=\"background-color: #FFFFFF;\">open</span> <span style=\"background-color: #6DB5F5;\">&lt;UNK&gt;</span> <span style=\"background-color: #6DB5F5;\">looked</span> <span style=\"background-color: #6DB5F5;\">like</span> <span style=\"background-color: #6DB5F5;\">watered</span> <span style=\"background-color: #6DB5F5;\">down</span> <span style=\"background-color: #6DB5F5;\">cola</span> <span style=\"background-color: #6DB5F5;\">head</span> <span style=\"background-color: #6DB5F5;\">fizzled</span> <span style=\"background-color: #6DB5F5;\">fast</span> <span style=\"background-color: #6DB5F5;\">not</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">good</span> <span style=\"background-color: #FFFFFF;\">sign</span> <span style=\"background-color: #FFFFFF;\">as</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">definitely</span> <span style=\"background-color: #FFFFFF;\">was</span> <span style=\"background-color: #FFFFFF;\">n</span> <span style=\"background-color: #FFFFFF;\">t</span> <span style=\"background-color: #FFFFFF;\">too</span> <span style=\"background-color: #FFFFFF;\">cold</span> <span style=\"background-color: #6DB5F5;\">hey</span> <span style=\"background-color: #6DB5F5;\">decent</span> <span style=\"background-color: #FFFFFF;\">lacing</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">s</span> <span style=\"background-color: #FFFFFF;\">really</span> <span style=\"background-color: #FFFFFF;\">trying</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">give</span> <span style=\"background-color: #FFFFFF;\">me</span> <span style=\"background-color: #FFFFFF;\">that</span> <span style=\"background-color: #FFFFFF;\">chocolate</span> <span style=\"background-color: #FFFFFF;\">coffee</span> <span style=\"background-color: #FFFFFF;\">stout</span> <span style=\"background-color: #FFFFFF;\">smell</span> <span style=\"background-color: #FFFFFF;\">here</span> <span style=\"background-color: #FFFFFF;\">but</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">s</span> <span style=\"background-color: #FFFFFF;\">muted</span> <span style=\"background-color: #FFFFFF;\">some</span> <span style=\"background-color: #FFFFFF;\">slight</span> <span style=\"background-color: #FFFFFF;\">coffe</span> <span style=\"background-color: #FFFFFF;\">toffee</span> <span style=\"background-color: #FFFFFF;\">taste</span> <span style=\"background-color: #FFFFFF;\">initially</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">hint</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">hop</span> <span style=\"background-color: #FFFFFF;\">bitterness</span> <span style=\"background-color: #FFFFFF;\">maybe</span> <span style=\"background-color: #FFFFFF;\">even</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">little</span> <span style=\"background-color: #FFFFFF;\">nutty</span> <span style=\"background-color: #FFFFFF;\">but</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">seems</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">be</span> <span style=\"background-color: #FFFFFF;\">out</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">balance</span> <span style=\"background-color: #FFFFFF;\">hence</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">blending</span> <span style=\"background-color: #FFFFFF;\">thing</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">guess</span> <span style=\"background-color: #FFFFFF;\">very</span> <span style=\"background-color: #FFFFFF;\">very</span> <span style=\"background-color: #FFFFFF;\">thin</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">watery</span> <span style=\"background-color: #FFFFFF;\">given</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">s</span> <span style=\"background-color: #FFFFFF;\">parent</span> <span style=\"background-color: #FFFFFF;\">ingredients</span> <span style=\"background-color: #FFFFFF;\">got</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">in</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">beers</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">world</span> <span style=\"background-color: #FFFFFF;\">pack</span> <span style=\"background-color: #FFFFFF;\">so</span> <span style=\"background-color: #FFFFFF;\">bonus</span> <span style=\"background-color: #FFFFFF;\">would</span> <span style=\"background-color: #FFFFFF;\">certainly</span> <span style=\"background-color: #FFFFFF;\">taste</span> <span style=\"background-color: #FFFFFF;\">fairy</span> <span style=\"background-color: #FFFFFF;\">exotic</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">macro</span> <span style=\"background-color: #FFFFFF;\">lager</span> <span style=\"background-color: #FFFFFF;\">person</span> <span style=\"background-color: #FFFFFF;\">but</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">wo</span> <span style=\"background-color: #FFFFFF;\">n</span> <span style=\"background-color: #FFFFFF;\">t</span> <span style=\"background-color: #FFFFFF;\">be</span> <span style=\"background-color: #FFFFFF;\">&lt;UNK&gt;</span> <span style=\"background-color: #FFFFFF;\">my</span> <span style=\"background-color: #FFFFFF;\">&lt;UNK&gt;</span> <span style=\"background-color: #FFFFFF;\">account</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">get</span> <span style=\"background-color: #FFFFFF;\">some</span> <span style=\"background-color: #FFFFFF;\">more</span> <span style=\"background-color: #FFFFFF;\">out</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">&lt;UNK&gt;</span> <span style=\"background-color: #FFFFFF;\">usa</span> <span style=\"background-color: #FFFFFF;\">anytime</span> <span style=\"background-color: #6DB5F5;\">soon</span> <span style=\"background-color: #6DB5F5;\"></span> <span style=\"background-color: #6DB5F5;\">&lt;PAD&gt;</span></p>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "expl_valid[121].visualize_in_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #FFFFFF;\">&lt;START&gt;</span> <span style=\"background-color: #FFFFFF;\">beer</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #6DB5F5;\">a</span> <span style=\"background-color: #6DB5F5;\">dark</span> <span style=\"background-color: #6DB5F5;\">dark</span> <span style=\"background-color: #6DB5F5;\">color</span> <span style=\"background-color: #6DB5F5;\">with</span> <span style=\"background-color: #6DB5F5;\">just</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">slightest</span> <span style=\"background-color: #FFFFFF;\">hints</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">ruby</span> <span style=\"background-color: #FFFFFF;\">at</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">edges</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #6DB5F5;\">a</span> <span style=\"background-color: #6DB5F5;\">coffee</span> <span style=\"background-color: #6DB5F5;\">colored</span> <span style=\"background-color: #6DB5F5;\">head</span> <span style=\"background-color: #6DB5F5;\">beautiful</span> <span style=\"background-color: #6DB5F5;\">to</span> <span style=\"background-color: #6DB5F5;\">look</span> <span style=\"background-color: #FFFFFF;\">at</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">almost</span> <span style=\"background-color: #FFFFFF;\">as</span> <span style=\"background-color: #FFFFFF;\">nice</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">drink</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">smell</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">coffee</span> <span style=\"background-color: #FFFFFF;\">cocoa</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">just</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">hint</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">caramel</span> <span style=\"background-color: #FFFFFF;\">or</span> <span style=\"background-color: #FFFFFF;\">toffee</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">an</span> <span style=\"background-color: #FFFFFF;\">underlying</span> <span style=\"background-color: #FFFFFF;\">alcohol</span> <span style=\"background-color: #FFFFFF;\">character</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">honey</span> <span style=\"background-color: #FFFFFF;\">sweetness</span> <span style=\"background-color: #FFFFFF;\">taste</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">very</span> <span style=\"background-color: #FFFFFF;\">similar</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">smell</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">coffee</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">cocoa</span> <span style=\"background-color: #FFFFFF;\">taking</span> <span style=\"background-color: #FFFFFF;\">center</span> <span style=\"background-color: #FFFFFF;\">stage</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">alcohol</span> <span style=\"background-color: #FFFFFF;\">almost</span> <span style=\"background-color: #FFFFFF;\">overpowering</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">toffee</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">caramel</span> <span style=\"background-color: #FFFFFF;\">notes</span> <span style=\"background-color: #FFFFFF;\">luckily</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">sweetness</span> <span style=\"background-color: #FFFFFF;\">helps</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">balance</span> <span style=\"background-color: #FFFFFF;\">that</span> <span style=\"background-color: #FFFFFF;\">out</span> <span style=\"background-color: #FFFFFF;\">mouthfeel</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #6DB5F5;\">good</span> <span style=\"background-color: #6DB5F5;\">nice</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #6DB5F5;\">thick</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">just</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">hint</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">stickiness</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">drinkability</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">n</span> <span style=\"background-color: #FFFFFF;\">t</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">best</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">was</span> <span style=\"background-color: #FFFFFF;\">actually</span> <span style=\"background-color: #FFFFFF;\">surprised</span> <span style=\"background-color: #FFFFFF;\">that</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">did</span> <span style=\"background-color: #FFFFFF;\">n</span> <span style=\"background-color: #FFFFFF;\">t</span> <span style=\"background-color: #FFFFFF;\">have</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">highest</span> <span style=\"background-color: #FFFFFF;\">abv</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">beers</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">had</span> <span style=\"background-color: #FFFFFF;\">at</span> <span style=\"background-color: #FFFFFF;\">dragonmead</span> <span style=\"background-color: #FFFFFF;\">that</span> <span style=\"background-color: #FFFFFF;\">being</span> <span style=\"background-color: #FFFFFF;\">said</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">was</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">perfectly</span> <span style=\"background-color: #FFFFFF;\">enjoyable</span> <span style=\"background-color: #FFFFFF;\">beer</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">d</span> <span style=\"background-color: #FFFFFF;\">jump</span> <span style=\"background-color: #FFFFFF;\">at</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">chance</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #6DB5F5;\">have</span> <span style=\"background-color: #6DB5F5;\">another</span> <span style=\"background-color: #6DB5F5;\"></span> <span style=\"background-color: #6DB5F5;\">&lt;PAD&gt;</span></p>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "expl_valid[888].visualize_in_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #FFFFFF;\">&lt;START&gt;</span> <span style=\"background-color: #FFFFFF;\">ten</span> <span style=\"background-color: #FFFFFF;\">fidy</span> <span style=\"background-color: #FFFFFF;\">another</span> <span style=\"background-color: #FFFFFF;\">thanks</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">&lt;UNK&gt;</span> <span style=\"background-color: #FFFFFF;\">for</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">trade</span> <span style=\"background-color: #FFFFFF;\">bod</span> <span style=\"background-color: #FFFFFF;\">17</span> <span style=\"background-color: #FFFFFF;\">pours</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">rusted</span> <span style=\"background-color: #FFFFFF;\">mahoghany</span> <span style=\"background-color: #6DB5F5;\">and</span> <span style=\"background-color: #6DB5F5;\">settles</span> <span style=\"background-color: #6DB5F5;\">jet</span> <span style=\"background-color: #6DB5F5;\">metal</span> <span style=\"background-color: #6DB5F5;\">black</span> <span style=\"background-color: #6DB5F5;\">a</span> <span style=\"background-color: #6DB5F5;\">fingers</span> <span style=\"background-color: #6DB5F5;\">worth</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #6DB5F5;\">burnt</span> <span style=\"background-color: #6DB5F5;\">caramel</span> <span style=\"background-color: #6DB5F5;\">head</span> <span style=\"background-color: #6DB5F5;\">sits</span> <span style=\"background-color: #6DB5F5;\">for</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">short</span> <span style=\"background-color: #FFFFFF;\">while</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">edges</span> <span style=\"background-color: #FFFFFF;\">leave</span> <span style=\"background-color: #FFFFFF;\">very</span> <span style=\"background-color: #FFFFFF;\">little</span> <span style=\"background-color: #FFFFFF;\">light</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">pass</span> <span style=\"background-color: #FFFFFF;\">through</span> <span style=\"background-color: #FFFFFF;\">spotty</span> <span style=\"background-color: #FFFFFF;\">lacing</span> <span style=\"background-color: #6DB5F5;\">clings</span> <span style=\"background-color: #6DB5F5;\">throughout</span> <span style=\"background-color: #6DB5F5;\">the</span> <span style=\"background-color: #FFFFFF;\">nose</span> <span style=\"background-color: #FFFFFF;\">brings</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">lot</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">milk</span> <span style=\"background-color: #FFFFFF;\">chocolate</span> <span style=\"background-color: #FFFFFF;\">that</span> <span style=\"background-color: #FFFFFF;\">has</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">bittering</span> <span style=\"background-color: #FFFFFF;\">end</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">roasted</span> <span style=\"background-color: #FFFFFF;\">malts</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">presence</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">alcohol</span> <span style=\"background-color: #FFFFFF;\">are</span> <span style=\"background-color: #FFFFFF;\">also</span> <span style=\"background-color: #FFFFFF;\">noted</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">taste</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">interesting</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">roasty</span> <span style=\"background-color: #FFFFFF;\">malts</span> <span style=\"background-color: #FFFFFF;\">bloom</span> <span style=\"background-color: #FFFFFF;\">but</span> <span style=\"background-color: #FFFFFF;\">an</span> <span style=\"background-color: #FFFFFF;\">annoying</span> <span style=\"background-color: #FFFFFF;\">metallic</span> <span style=\"background-color: #FFFFFF;\">taste</span> <span style=\"background-color: #FFFFFF;\">lingers</span> <span style=\"background-color: #FFFFFF;\">there</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">light</span> <span style=\"background-color: #FFFFFF;\">hop</span> <span style=\"background-color: #FFFFFF;\">presence</span> <span style=\"background-color: #FFFFFF;\">as</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">warms</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">flavors</span> <span style=\"background-color: #FFFFFF;\">intensify</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">mouthfeel</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">very</span> <span style=\"background-color: #FFFFFF;\">full</span> <span style=\"background-color: #FFFFFF;\">bodied</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">sits</span> <span style=\"background-color: #FFFFFF;\">like</span> <span style=\"background-color: #FFFFFF;\">an</span> <span style=\"background-color: #FFFFFF;\">&lt;UNK&gt;</span> <span style=\"background-color: #FFFFFF;\">creamy</span> <span style=\"background-color: #FFFFFF;\">feel</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">good</span> <span style=\"background-color: #FFFFFF;\">carbonation</span> <span style=\"background-color: #FFFFFF;\">overall</span> <span style=\"background-color: #FFFFFF;\">pretty</span> <span style=\"background-color: #FFFFFF;\">good</span> <span style=\"background-color: #FFFFFF;\">impy</span> <span style=\"background-color: #FFFFFF;\">stout</span> <span style=\"background-color: #FFFFFF;\">but</span> <span style=\"background-color: #FFFFFF;\">that</span> <span style=\"background-color: #FFFFFF;\">metal</span> <span style=\"background-color: #FFFFFF;\">taste</span> <span style=\"background-color: #FFFFFF;\">was</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">bit</span> <span style=\"background-color: #FFFFFF;\">off</span> <span style=\"background-color: #FFFFFF;\">putting</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">would</span> <span style=\"background-color: #FFFFFF;\">like</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">try</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">fresh</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">see</span> <span style=\"background-color: #FFFFFF;\">if</span> <span style=\"background-color: #FFFFFF;\">there</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #6DB5F5;\">a</span> <span style=\"background-color: #6DB5F5;\">difference</span> <span style=\"background-color: #6DB5F5;\"></span> <span style=\"background-color: #6DB5F5;\">&lt;PAD&gt;</span></p>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "expl_valid[999].visualize_in_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #FFFFFF;\">&lt;START&gt;</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #6DB5F5;\">a</span> <span style=\"background-color: #6DB5F5;\">wow</span> <span style=\"background-color: #6DB5F5;\">witbier</span> <span style=\"background-color: #6DB5F5;\">cloudy</span> <span style=\"background-color: #6DB5F5;\">yellow</span> <span style=\"background-color: #6DB5F5;\">with</span> <span style=\"background-color: #6DB5F5;\">tendencies</span> <span style=\"background-color: #6DB5F5;\">toward</span> <span style=\"background-color: #FFFFFF;\">something</span> <span style=\"background-color: #FFFFFF;\">darker</span> <span style=\"background-color: #6DB5F5;\">more</span> <span style=\"background-color: #6DB5F5;\">orange</span> <span style=\"background-color: #6DB5F5;\">the</span> <span style=\"background-color: #6DB5F5;\">head</span> <span style=\"background-color: #6DB5F5;\">is</span> <span style=\"background-color: #6DB5F5;\">a</span> <span style=\"background-color: #6DB5F5;\">little</span> <span style=\"background-color: #6DB5F5;\">flat</span> <span style=\"background-color: #FFFFFF;\">though</span> <span style=\"background-color: #FFFFFF;\">there</span> <span style=\"background-color: #FFFFFF;\">s</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">good</span> <span style=\"background-color: #FFFFFF;\">flowery</span> <span style=\"background-color: #FFFFFF;\">perfume</span> <span style=\"background-color: #FFFFFF;\">aroma</span> <span style=\"background-color: #FFFFFF;\">soft</span> <span style=\"background-color: #FFFFFF;\">citrus</span> <span style=\"background-color: #FFFFFF;\">light</span> <span style=\"background-color: #FFFFFF;\">coriander</span> <span style=\"background-color: #FFFFFF;\">come</span> <span style=\"background-color: #FFFFFF;\">up</span> <span style=\"background-color: #FFFFFF;\">front</span> <span style=\"background-color: #FFFFFF;\">wiht</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">good</span> <span style=\"background-color: #FFFFFF;\">dry</span> <span style=\"background-color: #FFFFFF;\">wheat</span> <span style=\"background-color: #FFFFFF;\">in</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">finish</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">little</span> <span style=\"background-color: #FFFFFF;\">heftier</span> <span style=\"background-color: #FFFFFF;\">than</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">supposed</span> <span style=\"background-color: #FFFFFF;\">style</span> <span style=\"background-color: #FFFFFF;\">stalwart</span> <span style=\"background-color: #FFFFFF;\">hoegaarden</span> <span style=\"background-color: #FFFFFF;\">but</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">soft</span> <span style=\"background-color: #FFFFFF;\">almost</span> <span style=\"background-color: #FFFFFF;\">creamy</span> <span style=\"background-color: #FFFFFF;\">mouthfeel</span> <span style=\"background-color: #FFFFFF;\">makes</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">surprisingly</span> <span style=\"background-color: #FFFFFF;\">satisfying</span> <span style=\"background-color: #FFFFFF;\">beer</span> <span style=\"background-color: #FFFFFF;\">without</span> <span style=\"background-color: #FFFFFF;\">being</span> <span style=\"background-color: #FFFFFF;\">heavy</span> <span style=\"background-color: #6DB5F5;\">tasty</span> <span style=\"background-color: #6DB5F5;\">beer</span> <span style=\"background-color: #6DB5F5;\"></span> <span style=\"background-color: #6DB5F5;\">&lt;PAD&gt;</span></p>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "expl_valid[333].visualize_in_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #6DB5F5;\">&lt;START&gt;</span> <span style=\"background-color: #6DB5F5;\">dark</span> <span style=\"background-color: #6DB5F5;\">black</span> <span style=\"background-color: #6DB5F5;\">with</span> <span style=\"background-color: #6DB5F5;\">creamy</span> <span style=\"background-color: #6DB5F5;\">tan</span> <span style=\"background-color: #6DB5F5;\">head</span> <span style=\"background-color: #6DB5F5;\">that</span> <span style=\"background-color: #6DB5F5;\">leaves</span> <span style=\"background-color: #6DB5F5;\">great</span> <span style=\"background-color: #6DB5F5;\">retention</span> <span style=\"background-color: #6DB5F5;\">and</span> <span style=\"background-color: #6DB5F5;\">foamy</span> <span style=\"background-color: #6DB5F5;\">lace</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">smell</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">roasty</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">burnt</span> <span style=\"background-color: #FFFFFF;\">sugar</span> <span style=\"background-color: #FFFFFF;\">edges</span> <span style=\"background-color: #FFFFFF;\">dark</span> <span style=\"background-color: #FFFFFF;\">chocolate</span> <span style=\"background-color: #FFFFFF;\">coffee</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">smoke</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">taste</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">ashy</span> <span style=\"background-color: #FFFFFF;\">too</span> <span style=\"background-color: #FFFFFF;\">much</span> <span style=\"background-color: #6DB5F5;\">black</span> <span style=\"background-color: #FFFFFF;\">patent</span> <span style=\"background-color: #6DB5F5;\">perhaps</span> <span style=\"background-color: #FFFFFF;\">others</span> <span style=\"background-color: #FFFFFF;\">enjoy</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">but</span> <span style=\"background-color: #FFFFFF;\">there</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">charcoal</span> <span style=\"background-color: #FFFFFF;\">burnt</span> <span style=\"background-color: #FFFFFF;\">taste</span> <span style=\"background-color: #FFFFFF;\">that</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">bit</span> <span style=\"background-color: #FFFFFF;\">much</span> <span style=\"background-color: #FFFFFF;\">for</span> <span style=\"background-color: #FFFFFF;\">me</span> <span style=\"background-color: #FFFFFF;\">smoky</span> <span style=\"background-color: #FFFFFF;\">bitter</span> <span style=\"background-color: #FFFFFF;\">chocolate</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">roasted</span> <span style=\"background-color: #FFFFFF;\">coffee</span> <span style=\"background-color: #FFFFFF;\">quite</span> <span style=\"background-color: #FFFFFF;\">roasty</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">ashy</span> <span style=\"background-color: #FFFFFF;\">tasting</span> <span style=\"background-color: #FFFFFF;\">strong</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">alcohol</span> <span style=\"background-color: #FFFFFF;\">peeking</span> <span style=\"background-color: #FFFFFF;\">through</span> <span style=\"background-color: #FFFFFF;\">overall</span> <span style=\"background-color: #FFFFFF;\">an</span> <span style=\"background-color: #6DB5F5;\">average</span> <span style=\"background-color: #6DB5F5;\">stout</span> <span style=\"background-color: #6DB5F5;\"></span> <span style=\"background-color: #6DB5F5;\">&lt;PAD&gt;</span></p>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "expl_valid[111].visualize_in_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #FFFFFF;\">&lt;START&gt;</span> <span style=\"background-color: #FFFFFF;\">reviewing</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">oaked</span> <span style=\"background-color: #FFFFFF;\">arrogant</span> <span style=\"background-color: #FFFFFF;\">bastard</span> <span style=\"background-color: #FFFFFF;\">ale</span> <span style=\"background-color: #FFFFFF;\">from</span> <span style=\"background-color: #FFFFFF;\">stone</span> <span style=\"background-color: #FFFFFF;\">brewing</span> <span style=\"background-color: #FFFFFF;\">company</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">hearty</span> <span style=\"background-color: #FFFFFF;\">thank</span> <span style=\"background-color: #FFFFFF;\">you</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">beeradvocate</span> <span style=\"background-color: #FFFFFF;\">user</span> <span style=\"background-color: #FFFFFF;\">funhog</span> <span style=\"background-color: #FFFFFF;\">for</span> <span style=\"background-color: #FFFFFF;\">hooking</span> <span style=\"background-color: #FFFFFF;\">me</span> <span style=\"background-color: #FFFFFF;\">up</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">one</span> <span style=\"background-color: #FFFFFF;\">score</span> <span style=\"background-color: #FFFFFF;\">appearance</span> <span style=\"background-color: #FFFFFF;\">pours</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #6DB5F5;\">dark</span> <span style=\"background-color: #6DB5F5;\">red</span> <span style=\"background-color: #6DB5F5;\">brown</span> <span style=\"background-color: #6DB5F5;\">color</span> <span style=\"background-color: #6DB5F5;\">with</span> <span style=\"background-color: #6DB5F5;\">plenty</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">opaque</span> <span style=\"background-color: #FFFFFF;\">ish</span> <span style=\"background-color: #6DB5F5;\">ruby</span> <span style=\"background-color: #6DB5F5;\">highlights</span> <span style=\"background-color: #6DB5F5;\">with</span> <span style=\"background-color: #6DB5F5;\">three</span> <span style=\"background-color: #6DB5F5;\">fingers</span> <span style=\"background-color: #6DB5F5;\">of</span> <span style=\"background-color: #6DB5F5;\">cream</span> <span style=\"background-color: #6DB5F5;\">colored</span> <span style=\"background-color: #6DB5F5;\">head</span> <span style=\"background-color: #6DB5F5;\">excellent</span> <span style=\"background-color: #FFFFFF;\">lacing</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">head</span> <span style=\"background-color: #FFFFFF;\">really</span> <span style=\"background-color: #FFFFFF;\">stick</span> <span style=\"background-color: #FFFFFF;\">around</span> <span style=\"background-color: #FFFFFF;\">if</span> <span style=\"background-color: #FFFFFF;\">not</span> <span style=\"background-color: #FFFFFF;\">apparent</span> <span style=\"background-color: #FFFFFF;\">by</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">photo</span> <span style=\"background-color: #FFFFFF;\">proprietary</span> <span style=\"background-color: #FFFFFF;\">5</span> <span style=\"background-color: #FFFFFF;\">smell</span> <span style=\"background-color: #FFFFFF;\">piney</span> <span style=\"background-color: #FFFFFF;\">citrusy</span> <span style=\"background-color: #FFFFFF;\">hops</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">oak</span> <span style=\"background-color: #FFFFFF;\">wood</span> <span style=\"background-color: #FFFFFF;\">up</span> <span style=\"background-color: #FFFFFF;\">front</span> <span style=\"background-color: #FFFFFF;\">creamy</span> <span style=\"background-color: #FFFFFF;\">chocolate</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">little</span> <span style=\"background-color: #FFFFFF;\">caramel</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">figs</span> <span style=\"background-color: #FFFFFF;\">oranges</span> <span style=\"background-color: #FFFFFF;\">tangerines</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">malts</span> <span style=\"background-color: #FFFFFF;\">5</span> <span style=\"background-color: #FFFFFF;\">taste</span> <span style=\"background-color: #FFFFFF;\">very</span> <span style=\"background-color: #FFFFFF;\">sweet</span> <span style=\"background-color: #FFFFFF;\">caramel</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">citrus</span> <span style=\"background-color: #FFFFFF;\">hoppy</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">toasted</span> <span style=\"background-color: #FFFFFF;\">maltiness</span> <span style=\"background-color: #FFFFFF;\">slightly</span> <span style=\"background-color: #FFFFFF;\">bitter</span> <span style=\"background-color: #FFFFFF;\">finish</span> <span style=\"background-color: #FFFFFF;\">5</span> <span style=\"background-color: #FFFFFF;\">mouthfeel</span> <span style=\"background-color: #FFFFFF;\">medium</span> <span style=\"background-color: #FFFFFF;\">bodied</span> <span style=\"background-color: #FFFFFF;\">oily</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">cream</span> <span style=\"background-color: #FFFFFF;\">low</span> <span style=\"background-color: #FFFFFF;\">carbonation</span> <span style=\"background-color: #FFFFFF;\">complements</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">viscosity</span> <span style=\"background-color: #FFFFFF;\">well</span> <span style=\"background-color: #FFFFFF;\">dry</span> <span style=\"background-color: #FFFFFF;\">bitter</span> <span style=\"background-color: #FFFFFF;\">finish</span> <span style=\"background-color: #FFFFFF;\">5</span> <span style=\"background-color: #FFFFFF;\">overall</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">very</span> <span style=\"background-color: #FFFFFF;\">solid</span> <span style=\"background-color: #FFFFFF;\">brew</span> <span style=\"background-color: #FFFFFF;\">but</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">original</span> <span style=\"background-color: #FFFFFF;\">version</span> <span style=\"background-color: #FFFFFF;\">arrogant</span> <span style=\"background-color: #FFFFFF;\">bastard</span> <span style=\"background-color: #FFFFFF;\">ale</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">better</span> <span style=\"background-color: #FFFFFF;\">in</span> <span style=\"background-color: #FFFFFF;\">my</span> <span style=\"background-color: #FFFFFF;\">opinion</span> <span style=\"background-color: #FFFFFF;\">double</span> <span style=\"background-color: #FFFFFF;\">bastard</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">even</span> <span style=\"background-color: #FFFFFF;\">better</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">beer</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">absolutely</span> <span style=\"background-color: #FFFFFF;\">worth</span> <span style=\"background-color: #FFFFFF;\">trying</span> <span style=\"background-color: #FFFFFF;\">but</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">six</span> <span style=\"background-color: #FFFFFF;\">pack</span> <span style=\"background-color: #FFFFFF;\">seems</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">bit</span> <span style=\"background-color: #FFFFFF;\">much</span> <span style=\"background-color: #FFFFFF;\">on</span> <span style=\"background-color: #FFFFFF;\">quantity</span> <span style=\"background-color: #FFFFFF;\">for</span> <span style=\"background-color: #FFFFFF;\">me</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">guess</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">have</span> <span style=\"background-color: #FFFFFF;\">some</span> <span style=\"background-color: #FFFFFF;\">extras</span> <span style=\"background-color: #FFFFFF;\">for</span> <span style=\"background-color: #FFFFFF;\">future</span> <span style=\"background-color: #FFFFFF;\">ba</span> <span style=\"background-color: #FFFFFF;\">trades</span> <span style=\"background-color: #FFFFFF;\">recommendation</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">can</span> <span style=\"background-color: #FFFFFF;\">certainly</span> <span style=\"background-color: #FFFFFF;\">recommend</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">one</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">both</span> <span style=\"background-color: #FFFFFF;\">beer</span> <span style=\"background-color: #FFFFFF;\">geeks</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">casual</span> <span style=\"background-color: #FFFFFF;\">beer</span> <span style=\"background-color: #FFFFFF;\">drinkers</span> <span style=\"background-color: #FFFFFF;\">as</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">flavors</span> <span style=\"background-color: #FFFFFF;\">are</span> <span style=\"background-color: #FFFFFF;\">pretty</span> <span style=\"background-color: #FFFFFF;\">solid</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">not</span> <span style=\"background-color: #FFFFFF;\">overwhelming</span> <span style=\"background-color: #FFFFFF;\">but</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">oaking</span> <span style=\"background-color: #FFFFFF;\">does</span> <span style=\"background-color: #FFFFFF;\">not</span> <span style=\"background-color: #FFFFFF;\">seem</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">add</span> <span style=\"background-color: #FFFFFF;\">enough</span> <span style=\"background-color: #FFFFFF;\">additional</span> <span style=\"background-color: #FFFFFF;\">character</span> <span style=\"background-color: #FFFFFF;\">flavor</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">justify</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">steep</span> <span style=\"background-color: #FFFFFF;\">price</span> <span style=\"background-color: #FFFFFF;\">jump</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">would</span> <span style=\"background-color: #FFFFFF;\">most</span> <span style=\"background-color: #FFFFFF;\">recommend</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">beer</span> <span style=\"background-color: #FFFFFF;\">as</span> <span style=\"background-color: #FFFFFF;\">one</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">add</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">mix</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">six</span> <span style=\"background-color: #FFFFFF;\">pairings</span> <span style=\"background-color: #FFFFFF;\">hamburger</span> <span style=\"background-color: #FFFFFF;\">cost</span> <span style=\"background-color: #FFFFFF;\">99</span> <span style=\"background-color: #FFFFFF;\">for</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #6DB5F5;\">six</span> <span style=\"background-color: #6DB5F5;\">pack</span> <span style=\"background-color: #6DB5F5;\"></span> <span style=\"background-color: #6DB5F5;\">&lt;PAD&gt;</span></p>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "expl_valid[100].visualize_in_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #FFFFFF;\">&lt;START&gt;</span> <span style=\"background-color: #FFFFFF;\">750ml</span> <span style=\"background-color: #FFFFFF;\">bottle</span> <span style=\"background-color: #FFFFFF;\">into</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">tulip</span> <span style=\"background-color: #FFFFFF;\">huge</span> <span style=\"background-color: #FFFFFF;\">thanks</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">kevin</span> <span style=\"background-color: #FFFFFF;\">for</span> <span style=\"background-color: #FFFFFF;\">sharing</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">ancient</span> <span style=\"background-color: #6DB5F5;\">oddity</span> <span style=\"background-color: #6DB5F5;\">a</span> <span style=\"background-color: #6DB5F5;\">muddy</span> <span style=\"background-color: #6DB5F5;\">magenta</span> <span style=\"background-color: #6DB5F5;\">brown</span> <span style=\"background-color: #6DB5F5;\">body</span> <span style=\"background-color: #6DB5F5;\">with</span> <span style=\"background-color: #6DB5F5;\">a</span> <span style=\"background-color: #6DB5F5;\">handful</span> <span style=\"background-color: #6DB5F5;\">of</span> <span style=\"background-color: #6DB5F5;\">off</span> <span style=\"background-color: #6DB5F5;\">white</span> <span style=\"background-color: #6DB5F5;\">bubbles</span> <span style=\"background-color: #6DB5F5;\">meh</span> <span style=\"background-color: #FFFFFF;\">s</span> <span style=\"background-color: #FFFFFF;\">old</span> <span style=\"background-color: #FFFFFF;\">musty</span> <span style=\"background-color: #6DB5F5;\">oaky</span> <span style=\"background-color: #6DB5F5;\">dirty</span> <span style=\"background-color: #FFFFFF;\">vaguely</span> <span style=\"background-color: #FFFFFF;\">reminiscent</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">tequila</span> <span style=\"background-color: #FFFFFF;\">in</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">very</span> <span style=\"background-color: #FFFFFF;\">weird</span> <span style=\"background-color: #FFFFFF;\">way</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">do</span> <span style=\"background-color: #FFFFFF;\">n</span> <span style=\"background-color: #FFFFFF;\">t</span> <span style=\"background-color: #FFFFFF;\">know</span> <span style=\"background-color: #FFFFFF;\">that</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">ve</span> <span style=\"background-color: #FFFFFF;\">ever</span> <span style=\"background-color: #FFFFFF;\">smelled</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">more</span> <span style=\"background-color: #FFFFFF;\">basementy</span> <span style=\"background-color: #FFFFFF;\">beer</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">kind</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">like</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">in</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">masochistic</span> <span style=\"background-color: #FFFFFF;\">way</span> <span style=\"background-color: #FFFFFF;\">t</span> <span style=\"background-color: #FFFFFF;\">like</span> <span style=\"background-color: #FFFFFF;\">liquid</span> <span style=\"background-color: #FFFFFF;\">dementia</span> <span style=\"background-color: #FFFFFF;\">so</span> <span style=\"background-color: #FFFFFF;\">so</span> <span style=\"background-color: #FFFFFF;\">old</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">yet</span> <span style=\"background-color: #FFFFFF;\">still</span> <span style=\"background-color: #FFFFFF;\">tasty</span> <span style=\"background-color: #FFFFFF;\">some</span> <span style=\"background-color: #FFFFFF;\">moderate</span> <span style=\"background-color: #FFFFFF;\">sourness</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">acidic</span> <span style=\"background-color: #FFFFFF;\">fruitiness</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">still</span> <span style=\"background-color: #FFFFFF;\">there</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">provide</span> <span style=\"background-color: #FFFFFF;\">at</span> <span style=\"background-color: #FFFFFF;\">least</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">hint</span> <span style=\"background-color: #FFFFFF;\">at</span> <span style=\"background-color: #FFFFFF;\">what</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">beer</span> <span style=\"background-color: #FFFFFF;\">used</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">be</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">dig</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">m</span> <span style=\"background-color: #FFFFFF;\">smooth</span> <span style=\"background-color: #FFFFFF;\">soft</span> <span style=\"background-color: #FFFFFF;\">amazingly</span> <span style=\"background-color: #FFFFFF;\">delicate</span> <span style=\"background-color: #FFFFFF;\">o</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">was</span> <span style=\"background-color: #FFFFFF;\">n</span> <span style=\"background-color: #FFFFFF;\">t</span> <span style=\"background-color: #FFFFFF;\">exactly</span> <span style=\"background-color: #FFFFFF;\">delicious</span> <span style=\"background-color: #FFFFFF;\">but</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">was</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">great</span> <span style=\"background-color: #FFFFFF;\">experience</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">wish</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">d</span> <span style=\"background-color: #FFFFFF;\">gotten</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">chance</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">taste</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">five</span> <span style=\"background-color: #FFFFFF;\">years</span> <span style=\"background-color: #6DB5F5;\">ago</span> <span style=\"background-color: #6DB5F5;\">cheers</span> <span style=\"background-color: #6DB5F5;\"></span> <span style=\"background-color: #6DB5F5;\">&lt;PAD&gt;</span></p>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "expl_valid[1021].visualize_in_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #FFFFFF;\">&lt;START&gt;</span> <span style=\"background-color: #6DB5F5;\">the</span> <span style=\"background-color: #6DB5F5;\">apperance</span> <span style=\"background-color: #6DB5F5;\">was</span> <span style=\"background-color: #6DB5F5;\">an</span> <span style=\"background-color: #6DB5F5;\">amber</span> <span style=\"background-color: #6DB5F5;\">dark</span> <span style=\"background-color: #6DB5F5;\">yellow</span> <span style=\"background-color: #6DB5F5;\">color</span> <span style=\"background-color: #6DB5F5;\">with</span> <span style=\"background-color: #6DB5F5;\">not</span> <span style=\"background-color: #6DB5F5;\">much</span> <span style=\"background-color: #6DB5F5;\">head</span> <span style=\"background-color: #6DB5F5;\">it</span> <span style=\"background-color: #6DB5F5;\">did</span> <span style=\"background-color: #6DB5F5;\">however</span> <span style=\"background-color: #FFFFFF;\">have</span> <span style=\"background-color: #6DB5F5;\">stuff</span> <span style=\"background-color: #6DB5F5;\">floating</span> <span style=\"background-color: #6DB5F5;\">in</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">m</span> <span style=\"background-color: #FFFFFF;\">not</span> <span style=\"background-color: #FFFFFF;\">certain</span> <span style=\"background-color: #FFFFFF;\">if</span> <span style=\"background-color: #FFFFFF;\">that</span> <span style=\"background-color: #FFFFFF;\">was</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">fault</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">manufacturer</span> <span style=\"background-color: #FFFFFF;\">or</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">fault</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">myself</span> <span style=\"background-color: #FFFFFF;\">for</span> <span style=\"background-color: #FFFFFF;\">trusting</span> <span style=\"background-color: #FFFFFF;\">my</span> <span style=\"background-color: #FFFFFF;\">friends</span> <span style=\"background-color: #FFFFFF;\">around</span> <span style=\"background-color: #FFFFFF;\">my</span> <span style=\"background-color: #FFFFFF;\">&lt;UNK&gt;</span> <span style=\"background-color: #FFFFFF;\">drink</span> <span style=\"background-color: #FFFFFF;\">any</span> <span style=\"background-color: #FFFFFF;\">who</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">smell</span> <span style=\"background-color: #FFFFFF;\">was</span> <span style=\"background-color: #FFFFFF;\">bellow</span> <span style=\"background-color: #FFFFFF;\">average</span> <span style=\"background-color: #6DB5F5;\">although</span> <span style=\"background-color: #FFFFFF;\">not</span> <span style=\"background-color: #FFFFFF;\">always</span> <span style=\"background-color: #FFFFFF;\">clearly</span> <span style=\"background-color: #FFFFFF;\">present</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">taste</span> <span style=\"background-color: #FFFFFF;\">was</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">sweet</span> <span style=\"background-color: #FFFFFF;\">sour</span> <span style=\"background-color: #FFFFFF;\">mix</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">main</span> <span style=\"background-color: #FFFFFF;\">taste</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">bitterness</span> <span style=\"background-color: #FFFFFF;\">mouthfeel</span> <span style=\"background-color: #FFFFFF;\">was</span> <span style=\"background-color: #FFFFFF;\">smooth</span> <span style=\"background-color: #FFFFFF;\">esque</span> <span style=\"background-color: #FFFFFF;\">drinkability</span> <span style=\"background-color: #FFFFFF;\">was</span> <span style=\"background-color: #FFFFFF;\">average</span> <span style=\"background-color: #FFFFFF;\">but</span> <span style=\"background-color: #FFFFFF;\">seeing</span> <span style=\"background-color: #FFFFFF;\">as</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">m</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">big</span> <span style=\"background-color: #FFFFFF;\">time</span> <span style=\"background-color: #FFFFFF;\">&lt;UNK&gt;</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">seeing</span> <span style=\"background-color: #FFFFFF;\">as</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">s</span> <span style=\"background-color: #FFFFFF;\">what</span> <span style=\"background-color: #FFFFFF;\">my</span> <span style=\"background-color: #FFFFFF;\">friends</span> <span style=\"background-color: #FFFFFF;\">have</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">will</span> <span style=\"background-color: #FFFFFF;\">most</span> <span style=\"background-color: #FFFFFF;\">likely</span> <span style=\"background-color: #FFFFFF;\">be</span> <span style=\"background-color: #FFFFFF;\">having</span> <span style=\"background-color: #FFFFFF;\">another</span> <span style=\"background-color: #6DB5F5;\">very</span> <span style=\"background-color: #FFFFFF;\">soon</span> <span style=\"background-color: #FFFFFF;\"></span> <span style=\"background-color: #FFFFFF;\">&lt;PAD&gt;</span></p>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "expl_valid[9999].visualize_in_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #6DB5F5;\">&lt;START&gt;</span> <span style=\"background-color: #6DB5F5;\">pale</span> <span style=\"background-color: #6DB5F5;\">gold</span> <span style=\"background-color: #6DB5F5;\">with</span> <span style=\"background-color: #6DB5F5;\">a</span> <span style=\"background-color: #6DB5F5;\">thin</span> <span style=\"background-color: #6DB5F5;\">film</span> <span style=\"background-color: #6DB5F5;\">around</span> <span style=\"background-color: #6DB5F5;\">the</span> <span style=\"background-color: #6DB5F5;\">edge</span> <span style=\"background-color: #6DB5F5;\">some</span> <span style=\"background-color: #6DB5F5;\">lacing</span> <span style=\"background-color: #6DB5F5;\">looks</span> <span style=\"background-color: #6DB5F5;\">very</span> <span style=\"background-color: #6DB5F5;\">flat</span> <span style=\"background-color: #6DB5F5;\">and</span> <span style=\"background-color: #6DB5F5;\">insipid</span> <span style=\"background-color: #FFFFFF;\">no</span> <span style=\"background-color: #FFFFFF;\">carbonation</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">do</span> <span style=\"background-color: #FFFFFF;\">n</span> <span style=\"background-color: #FFFFFF;\">t</span> <span style=\"background-color: #FFFFFF;\">hold</span> <span style=\"background-color: #FFFFFF;\">out</span> <span style=\"background-color: #6DB5F5;\">much</span> <span style=\"background-color: #6DB5F5;\">hope</span> <span style=\"background-color: #FFFFFF;\">very</span> <span style=\"background-color: #FFFFFF;\">unpleasant</span> <span style=\"background-color: #FFFFFF;\">sticky</span> <span style=\"background-color: #FFFFFF;\">rice</span> <span style=\"background-color: #FFFFFF;\">nose</span> <span style=\"background-color: #FFFFFF;\">lots</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">nothingness</span> <span style=\"background-color: #FFFFFF;\">as</span> <span style=\"background-color: #FFFFFF;\">well</span> <span style=\"background-color: #FFFFFF;\">sweet</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">very</span> <span style=\"background-color: #6DB5F5;\">light</span> <span style=\"background-color: #FFFFFF;\">sickly</span> <span style=\"background-color: #FFFFFF;\">note</span> <span style=\"background-color: #FFFFFF;\">but</span> <span style=\"background-color: #FFFFFF;\">do</span> <span style=\"background-color: #FFFFFF;\">n</span> <span style=\"background-color: #FFFFFF;\">t</span> <span style=\"background-color: #FFFFFF;\">get</span> <span style=\"background-color: #FFFFFF;\">me</span> <span style=\"background-color: #FFFFFF;\">wrong</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">s</span> <span style=\"background-color: #FFFFFF;\">incredibly</span> <span style=\"background-color: #FFFFFF;\">bland</span> <span style=\"background-color: #FFFFFF;\">blech</span> <span style=\"background-color: #FFFFFF;\">thin</span> <span style=\"background-color: #FFFFFF;\">but</span> <span style=\"background-color: #FFFFFF;\">fortunately</span> <span style=\"background-color: #FFFFFF;\">not</span> <span style=\"background-color: #FFFFFF;\">overly</span> <span style=\"background-color: #FFFFFF;\">sweet</span> <span style=\"background-color: #FFFFFF;\">on</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">palate</span> <span style=\"background-color: #FFFFFF;\">quite</span> <span style=\"background-color: #FFFFFF;\">clean</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">dry</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">light</span> <span style=\"background-color: #FFFFFF;\">lingering</span> <span style=\"background-color: #FFFFFF;\">bitterness</span> <span style=\"background-color: #FFFFFF;\">mouthfeel</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">quite</span> <span style=\"background-color: #FFFFFF;\">crisp</span> <span style=\"background-color: #FFFFFF;\">which</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">blessing</span> <span style=\"background-color: #FFFFFF;\">no</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">s</span> <span style=\"background-color: #FFFFFF;\">not</span> <span style=\"background-color: #FFFFFF;\">great</span> <span style=\"background-color: #FFFFFF;\">but</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">was</span> <span style=\"background-color: #FFFFFF;\">expecting</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">lot</span> <span style=\"background-color: #FFFFFF;\">lot</span> <span style=\"background-color: #FFFFFF;\">worse</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">s</span> <span style=\"background-color: #FFFFFF;\">really</span> <span style=\"background-color: #FFFFFF;\">not</span> <span style=\"background-color: #FFFFFF;\">that</span> <span style=\"background-color: #FFFFFF;\">bad</span> <span style=\"background-color: #FFFFFF;\">when</span> <span style=\"background-color: #FFFFFF;\">you</span> <span style=\"background-color: #FFFFFF;\">get</span> <span style=\"background-color: #FFFFFF;\">down</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">s</span> <span style=\"background-color: #FFFFFF;\">not</span> <span style=\"background-color: #FFFFFF;\">amazing</span> <span style=\"background-color: #FFFFFF;\">but</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">s</span> <span style=\"background-color: #FFFFFF;\">pretty</span> <span style=\"background-color: #FFFFFF;\">clean</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">light</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">guess</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">m</span> <span style=\"background-color: #FFFFFF;\">just</span> <span style=\"background-color: #FFFFFF;\">pleased</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">does</span> <span style=\"background-color: #FFFFFF;\">n</span> <span style=\"background-color: #FFFFFF;\">t</span> <span style=\"background-color: #FFFFFF;\">have</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">sweet</span> <span style=\"background-color: #FFFFFF;\">sickly</span> <span style=\"background-color: #FFFFFF;\">character</span> <span style=\"background-color: #FFFFFF;\">promised</span> <span style=\"background-color: #FFFFFF;\">on</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">nose</span> <span style=\"background-color: #FFFFFF;\"></span> <span style=\"background-color: #FFFFFF;\">&lt;PAD&gt;</span></p>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "expl_valid[7676].visualize_in_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #FFFFFF;\">&lt;START&gt;</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">was</span> <span style=\"background-color: #FFFFFF;\">actually</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">little</span> <span style=\"background-color: #FFFFFF;\">surprised</span> <span style=\"background-color: #FFFFFF;\">by</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">one</span> <span style=\"background-color: #FFFFFF;\">surprised</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">was</span> <span style=\"background-color: #FFFFFF;\">not</span> <span style=\"background-color: #FFFFFF;\">vile</span> <span style=\"background-color: #6DB5F5;\">pours</span> <span style=\"background-color: #6DB5F5;\">a</span> <span style=\"background-color: #6DB5F5;\">clear</span> <span style=\"background-color: #6DB5F5;\">gold</span> <span style=\"background-color: #6DB5F5;\">color</span> <span style=\"background-color: #6DB5F5;\">with</span> <span style=\"background-color: #6DB5F5;\">a</span> <span style=\"background-color: #6DB5F5;\">thin</span> <span style=\"background-color: #6DB5F5;\">white</span> <span style=\"background-color: #6DB5F5;\">head</span> <span style=\"background-color: #6DB5F5;\">no</span> <span style=\"background-color: #6DB5F5;\">real</span> <span style=\"background-color: #6DB5F5;\">lacing</span> <span style=\"background-color: #6DB5F5;\">to</span> <span style=\"background-color: #6DB5F5;\">speak</span> <span style=\"background-color: #FFFFFF;\">of</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #6DB5F5;\">the</span> <span style=\"background-color: #6DB5F5;\">head</span> <span style=\"background-color: #6DB5F5;\">was</span> <span style=\"background-color: #6DB5F5;\">short</span> <span style=\"background-color: #6DB5F5;\">lived</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">aroma</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">lightly</span> <span style=\"background-color: #FFFFFF;\">sweet</span> <span style=\"background-color: #FFFFFF;\">which</span> <span style=\"background-color: #FFFFFF;\">was</span> <span style=\"background-color: #FFFFFF;\">another</span> <span style=\"background-color: #FFFFFF;\">surprise</span> <span style=\"background-color: #FFFFFF;\">light</span> <span style=\"background-color: #FFFFFF;\">bodied</span> <span style=\"background-color: #FFFFFF;\">with</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">barely</span> <span style=\"background-color: #FFFFFF;\">average</span> <span style=\"background-color: #FFFFFF;\">hops</span> <span style=\"background-color: #FFFFFF;\">flavor</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">finish</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">little</span> <span style=\"background-color: #FFFFFF;\">sweet</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">little</span> <span style=\"background-color: #FFFFFF;\">fruity</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">n</span> <span style=\"background-color: #FFFFFF;\">t</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">beer</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">would</span> <span style=\"background-color: #FFFFFF;\">seek</span> <span style=\"background-color: #FFFFFF;\">out</span> <span style=\"background-color: #FFFFFF;\">again</span> <span style=\"background-color: #FFFFFF;\">but</span> <span style=\"background-color: #FFFFFF;\">i</span> <span style=\"background-color: #FFFFFF;\">would</span> <span style=\"background-color: #FFFFFF;\">drink</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">in</span> <span style=\"background-color: #FFFFFF;\">korea</span> <span style=\"background-color: #FFFFFF;\">over</span> <span style=\"background-color: #FFFFFF;\">a</span> <span style=\"background-color: #FFFFFF;\">bud</span> <span style=\"background-color: #FFFFFF;\"></span> <span style=\"background-color: #FFFFFF;\">&lt;PAD&gt;</span></p>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "expl_valid[6767].visualize_in_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "        <p style=\"text-align: center\">\n",
       "            Text\n",
       "        </p>\n",
       "        <div style=\"border: 1px solid black;\">\n",
       "            <p style=\"font-size:14px;; margin-left: 1%; margin-right: 1%;\"><span style=\"background-color: #FFFFFF;\">&lt;START&gt;</span> <span style=\"background-color: #FFFFFF;\">pours</span> <span style=\"background-color: #6DB5F5;\">a</span> <span style=\"background-color: #6DB5F5;\">clear</span> <span style=\"background-color: #6DB5F5;\">deep</span> <span style=\"background-color: #6DB5F5;\">red</span> <span style=\"background-color: #6DB5F5;\">brown</span> <span style=\"background-color: #6DB5F5;\">with</span> <span style=\"background-color: #6DB5F5;\">a</span> <span style=\"background-color: #6DB5F5;\">big</span> <span style=\"background-color: #6DB5F5;\">white</span> <span style=\"background-color: #6DB5F5;\">head</span> <span style=\"background-color: #6DB5F5;\">malty</span> <span style=\"background-color: #6DB5F5;\">sweet</span> <span style=\"background-color: #FFFFFF;\">no</span> <span style=\"background-color: #FFFFFF;\">major</span> <span style=\"background-color: #FFFFFF;\">flavors</span> <span style=\"background-color: #FFFFFF;\">stand</span> <span style=\"background-color: #FFFFFF;\">out</span> <span style=\"background-color: #FFFFFF;\">though</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #FFFFFF;\">slightly</span> <span style=\"background-color: #FFFFFF;\">toasty</span> <span style=\"background-color: #FFFFFF;\">hops</span> <span style=\"background-color: #FFFFFF;\">are</span> <span style=\"background-color: #FFFFFF;\">clean</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">mellow</span> <span style=\"background-color: #FFFFFF;\">they</span> <span style=\"background-color: #FFFFFF;\">only</span> <span style=\"background-color: #FFFFFF;\">come</span> <span style=\"background-color: #FFFFFF;\">in</span> <span style=\"background-color: #FFFFFF;\">near</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">end</span> <span style=\"background-color: #FFFFFF;\">and</span> <span style=\"background-color: #FFFFFF;\">help</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">balance</span> <span style=\"background-color: #FFFFFF;\">the</span> <span style=\"background-color: #FFFFFF;\">beer</span> <span style=\"background-color: #FFFFFF;\">this</span> <span style=\"background-color: #FFFFFF;\">is</span> <span style=\"background-color: #6DB5F5;\">a</span> <span style=\"background-color: #6DB5F5;\">solid</span> <span style=\"background-color: #6DB5F5;\">simple</span> <span style=\"background-color: #6DB5F5;\">brown</span> <span style=\"background-color: #FFFFFF;\">it</span> <span style=\"background-color: #FFFFFF;\">s</span> <span style=\"background-color: #FFFFFF;\">so</span> <span style=\"background-color: #FFFFFF;\">great</span> <span style=\"background-color: #FFFFFF;\">to</span> <span style=\"background-color: #FFFFFF;\">finally</span> <span style=\"background-color: #FFFFFF;\">see</span> <span style=\"background-color: #FFFFFF;\">organic</span> <span style=\"background-color: #FFFFFF;\">beer</span> <span style=\"background-color: #FFFFFF;\">in</span> <span style=\"background-color: #6DB5F5;\">the</span> <span style=\"background-color: #6DB5F5;\">store</span> <span style=\"background-color: #6DB5F5;\"></span> <span style=\"background-color: #6DB5F5;\">&lt;PAD&gt;</span></p>\n",
       "        </div>\n",
       "        \n",
       "    </div>\n",
       "    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "expl_valid[3131].visualize_in_notebook()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "my_jup_kernel",
   "language": "python",
   "name": "my_jup_kernel"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "vscode": {
   "interpreter": {
    "hash": "77738023c946bcedeeb6a5c983bcfb7849325694ebd87acbc9267f45e9f4af48"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
